2023-02-11 04:00:08,388 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3 2023-02-11 04:00:08,401 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestMobRestoreSnapshotFromClientSimple timeout: 13 mins 2023-02-11 04:00:08,414 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-02-11 04:00:08,415 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/cluster_b673684c-6f28-f6ab-2a99-eadf0e1a7de2, deleteOnExit=true 2023-02-11 04:00:08,415 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-02-11 04:00:08,416 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/test.cache.data in system properties and HBase conf 2023-02-11 04:00:08,416 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/hadoop.tmp.dir in system properties and HBase conf 2023-02-11 04:00:08,416 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/hadoop.log.dir in system properties and HBase conf 2023-02-11 04:00:08,417 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/mapreduce.cluster.local.dir in system properties and HBase conf 2023-02-11 04:00:08,417 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-02-11 04:00:08,417 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-02-11 04:00:08,524 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-02-11 04:00:08,911 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-02-11 04:00:08,915 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-02-11 04:00:08,915 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-02-11 04:00:08,915 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-02-11 04:00:08,916 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-11 04:00:08,916 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-02-11 04:00:08,916 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-02-11 04:00:08,916 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-11 04:00:08,917 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-11 04:00:08,917 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-02-11 04:00:08,917 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/nfs.dump.dir in system properties and HBase conf 2023-02-11 04:00:08,917 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/java.io.tmpdir in system properties and HBase conf 2023-02-11 04:00:08,918 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-11 04:00:08,918 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-02-11 04:00:08,918 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-02-11 04:00:09,411 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-11 04:00:09,415 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-11 04:00:09,690 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-02-11 04:00:09,848 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-02-11 04:00:09,863 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-11 04:00:09,900 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-02-11 04:00:09,931 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/java.io.tmpdir/Jetty_localhost_46537_hdfs____.q3jrnj/webapp 2023-02-11 04:00:10,056 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46537 2023-02-11 04:00:10,092 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-11 04:00:10,092 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-11 04:00:10,502 WARN [Listener at localhost/37271] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-11 04:00:10,600 WARN [Listener at localhost/37271] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-11 04:00:10,617 WARN [Listener at localhost/37271] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-11 04:00:10,623 INFO [Listener at localhost/37271] log.Slf4jLog(67): jetty-6.1.26 2023-02-11 04:00:10,627 INFO [Listener at localhost/37271] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/java.io.tmpdir/Jetty_localhost_38921_datanode____.g19812/webapp 2023-02-11 04:00:10,722 INFO [Listener at localhost/37271] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:38921 2023-02-11 04:00:11,006 WARN [Listener at localhost/44183] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-11 04:00:11,018 WARN [Listener at localhost/44183] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-11 04:00:11,021 WARN [Listener at localhost/44183] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-11 04:00:11,023 INFO [Listener at localhost/44183] log.Slf4jLog(67): jetty-6.1.26 2023-02-11 04:00:11,027 INFO [Listener at localhost/44183] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/java.io.tmpdir/Jetty_localhost_34977_datanode____hulnsv/webapp 2023-02-11 04:00:11,124 INFO [Listener at localhost/44183] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34977 2023-02-11 04:00:11,135 WARN [Listener at localhost/34411] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-11 04:00:11,154 WARN [Listener at localhost/34411] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-11 04:00:11,156 WARN [Listener at localhost/34411] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-11 04:00:11,158 INFO [Listener at localhost/34411] log.Slf4jLog(67): jetty-6.1.26 2023-02-11 04:00:11,163 INFO [Listener at localhost/34411] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/java.io.tmpdir/Jetty_localhost_34321_datanode____a7gk60/webapp 2023-02-11 04:00:11,278 INFO [Listener at localhost/34411] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34321 2023-02-11 04:00:11,291 WARN [Listener at localhost/33753] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-11 04:00:11,481 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xff29d701a98e87ac: Processing first storage report for DS-e59e7235-d091-4888-a8ac-54b667a6e5b2 from datanode 9cf11592-7e5f-4e4e-a42e-40b00afd1a4c 2023-02-11 04:00:11,483 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xff29d701a98e87ac: from storage DS-e59e7235-d091-4888-a8ac-54b667a6e5b2 node DatanodeRegistration(127.0.0.1:34161, datanodeUuid=9cf11592-7e5f-4e4e-a42e-40b00afd1a4c, infoPort=40209, infoSecurePort=0, ipcPort=33753, storageInfo=lv=-57;cid=testClusterID;nsid=1558218879;c=1676088009489), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-02-11 04:00:11,483 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x3564cc36cdfe2ee8: Processing first storage report for DS-7f7ff339-c799-4e70-9519-662960207563 from datanode e749fdb3-195e-48c9-9e67-ba30e7a3d54e 2023-02-11 04:00:11,483 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x3564cc36cdfe2ee8: from storage DS-7f7ff339-c799-4e70-9519-662960207563 node DatanodeRegistration(127.0.0.1:37613, datanodeUuid=e749fdb3-195e-48c9-9e67-ba30e7a3d54e, infoPort=39779, infoSecurePort=0, ipcPort=34411, storageInfo=lv=-57;cid=testClusterID;nsid=1558218879;c=1676088009489), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 04:00:11,483 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xd830e829b02e12e7: Processing first storage report for DS-8f0e379d-716c-4016-9c93-5a6633025c44 from datanode 45440c1d-4b08-4665-a898-afaa9522b4b6 2023-02-11 04:00:11,483 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xd830e829b02e12e7: from storage DS-8f0e379d-716c-4016-9c93-5a6633025c44 node DatanodeRegistration(127.0.0.1:34349, datanodeUuid=45440c1d-4b08-4665-a898-afaa9522b4b6, infoPort=37119, infoSecurePort=0, ipcPort=44183, storageInfo=lv=-57;cid=testClusterID;nsid=1558218879;c=1676088009489), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 04:00:11,484 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xff29d701a98e87ac: Processing first storage report for DS-3c54f692-b19e-4a1b-8e9e-f772acfd8335 from datanode 9cf11592-7e5f-4e4e-a42e-40b00afd1a4c 2023-02-11 04:00:11,484 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xff29d701a98e87ac: from storage DS-3c54f692-b19e-4a1b-8e9e-f772acfd8335 node DatanodeRegistration(127.0.0.1:34161, datanodeUuid=9cf11592-7e5f-4e4e-a42e-40b00afd1a4c, infoPort=40209, infoSecurePort=0, ipcPort=33753, storageInfo=lv=-57;cid=testClusterID;nsid=1558218879;c=1676088009489), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 04:00:11,484 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x3564cc36cdfe2ee8: Processing first storage report for DS-90a2e763-165b-4645-8f09-4a9208d767bf from datanode e749fdb3-195e-48c9-9e67-ba30e7a3d54e 2023-02-11 04:00:11,484 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x3564cc36cdfe2ee8: from storage DS-90a2e763-165b-4645-8f09-4a9208d767bf node DatanodeRegistration(127.0.0.1:37613, datanodeUuid=e749fdb3-195e-48c9-9e67-ba30e7a3d54e, infoPort=39779, infoSecurePort=0, ipcPort=34411, storageInfo=lv=-57;cid=testClusterID;nsid=1558218879;c=1676088009489), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 04:00:11,484 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xd830e829b02e12e7: Processing first storage report for DS-b2239a29-cd5e-4211-849b-c95d9c3eb88c from datanode 45440c1d-4b08-4665-a898-afaa9522b4b6 2023-02-11 04:00:11,484 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xd830e829b02e12e7: from storage DS-b2239a29-cd5e-4211-849b-c95d9c3eb88c node DatanodeRegistration(127.0.0.1:34349, datanodeUuid=45440c1d-4b08-4665-a898-afaa9522b4b6, infoPort=37119, infoSecurePort=0, ipcPort=44183, storageInfo=lv=-57;cid=testClusterID;nsid=1558218879;c=1676088009489), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 04:00:11,674 DEBUG [Listener at localhost/33753] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3 2023-02-11 04:00:11,735 INFO [Listener at localhost/33753] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/cluster_b673684c-6f28-f6ab-2a99-eadf0e1a7de2/zookeeper_0, clientPort=63425, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/cluster_b673684c-6f28-f6ab-2a99-eadf0e1a7de2/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/cluster_b673684c-6f28-f6ab-2a99-eadf0e1a7de2/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-02-11 04:00:11,751 INFO [Listener at localhost/33753] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=63425 2023-02-11 04:00:11,762 INFO [Listener at localhost/33753] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 04:00:11,764 INFO [Listener at localhost/33753] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 04:00:12,429 INFO [Listener at localhost/33753] util.FSUtils(479): Created version file at hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6 with version=8 2023-02-11 04:00:12,429 INFO [Listener at localhost/33753] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/hbase-staging 2023-02-11 04:00:12,726 INFO [Listener at localhost/33753] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-11 04:00:13,185 INFO [Listener at localhost/33753] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=18 2023-02-11 04:00:13,216 INFO [Listener at localhost/33753] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:13,216 INFO [Listener at localhost/33753] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:13,217 INFO [Listener at localhost/33753] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-11 04:00:13,217 INFO [Listener at localhost/33753] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:13,217 INFO [Listener at localhost/33753] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-11 04:00:13,353 INFO [Listener at localhost/33753] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-11 04:00:13,422 DEBUG [Listener at localhost/33753] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-02-11 04:00:13,514 INFO [Listener at localhost/33753] ipc.NettyRpcServer(120): Bind to /172.31.14.131:41713 2023-02-11 04:00:13,524 INFO [Listener at localhost/33753] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 04:00:13,526 INFO [Listener at localhost/33753] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 04:00:13,546 INFO [Listener at localhost/33753] zookeeper.RecoverableZooKeeper(93): Process identifier=master:41713 connecting to ZooKeeper ensemble=127.0.0.1:63425 2023-02-11 04:00:13,584 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): master:417130x0, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-11 04:00:13,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:41713-0x1004189558d0000 connected 2023-02-11 04:00:13,610 DEBUG [Listener at localhost/33753] zookeeper.ZKUtil(164): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 04:00:13,611 DEBUG [Listener at localhost/33753] zookeeper.ZKUtil(164): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 04:00:13,614 DEBUG [Listener at localhost/33753] zookeeper.ZKUtil(164): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-11 04:00:13,621 DEBUG [Listener at localhost/33753] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=41713 2023-02-11 04:00:13,622 DEBUG [Listener at localhost/33753] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=41713 2023-02-11 04:00:13,622 DEBUG [Listener at localhost/33753] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=41713 2023-02-11 04:00:13,623 DEBUG [Listener at localhost/33753] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=41713 2023-02-11 04:00:13,623 DEBUG [Listener at localhost/33753] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=41713 2023-02-11 04:00:13,629 INFO [Listener at localhost/33753] master.HMaster(439): hbase.rootdir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6, hbase.cluster.distributed=false 2023-02-11 04:00:13,695 INFO [Listener at localhost/33753] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=18 2023-02-11 04:00:13,695 INFO [Listener at localhost/33753] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:13,696 INFO [Listener at localhost/33753] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:13,696 INFO [Listener at localhost/33753] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-11 04:00:13,696 INFO [Listener at localhost/33753] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:13,696 INFO [Listener at localhost/33753] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-11 04:00:13,701 INFO [Listener at localhost/33753] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-11 04:00:13,704 INFO [Listener at localhost/33753] ipc.NettyRpcServer(120): Bind to /172.31.14.131:46851 2023-02-11 04:00:13,706 INFO [Listener at localhost/33753] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-11 04:00:13,712 INFO [Listener at localhost/33753] mob.MobFileCache(124): MobFileCache disabled 2023-02-11 04:00:13,714 INFO [Listener at localhost/33753] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 04:00:13,716 INFO [Listener at localhost/33753] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 04:00:13,718 INFO [Listener at localhost/33753] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:46851 connecting to ZooKeeper ensemble=127.0.0.1:63425 2023-02-11 04:00:13,721 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): regionserver:468510x0, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-11 04:00:13,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:46851-0x1004189558d0001 connected 2023-02-11 04:00:13,723 DEBUG [Listener at localhost/33753] zookeeper.ZKUtil(164): regionserver:46851-0x1004189558d0001, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 04:00:13,724 DEBUG [Listener at localhost/33753] zookeeper.ZKUtil(164): regionserver:46851-0x1004189558d0001, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 04:00:13,725 DEBUG [Listener at localhost/33753] zookeeper.ZKUtil(164): regionserver:46851-0x1004189558d0001, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-11 04:00:13,725 DEBUG [Listener at localhost/33753] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46851 2023-02-11 04:00:13,726 DEBUG [Listener at localhost/33753] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46851 2023-02-11 04:00:13,726 DEBUG [Listener at localhost/33753] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46851 2023-02-11 04:00:13,726 DEBUG [Listener at localhost/33753] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46851 2023-02-11 04:00:13,727 DEBUG [Listener at localhost/33753] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46851 2023-02-11 04:00:13,740 INFO [Listener at localhost/33753] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=18 2023-02-11 04:00:13,740 INFO [Listener at localhost/33753] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:13,741 INFO [Listener at localhost/33753] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:13,741 INFO [Listener at localhost/33753] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-11 04:00:13,741 INFO [Listener at localhost/33753] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:13,742 INFO [Listener at localhost/33753] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-11 04:00:13,742 INFO [Listener at localhost/33753] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-11 04:00:13,743 INFO [Listener at localhost/33753] ipc.NettyRpcServer(120): Bind to /172.31.14.131:40829 2023-02-11 04:00:13,744 INFO [Listener at localhost/33753] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-11 04:00:13,745 INFO [Listener at localhost/33753] mob.MobFileCache(124): MobFileCache disabled 2023-02-11 04:00:13,746 INFO [Listener at localhost/33753] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 04:00:13,748 INFO [Listener at localhost/33753] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 04:00:13,750 INFO [Listener at localhost/33753] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:40829 connecting to ZooKeeper ensemble=127.0.0.1:63425 2023-02-11 04:00:13,755 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): regionserver:408290x0, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-11 04:00:13,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:40829-0x1004189558d0002 connected 2023-02-11 04:00:13,756 DEBUG [Listener at localhost/33753] zookeeper.ZKUtil(164): regionserver:40829-0x1004189558d0002, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 04:00:13,756 DEBUG [Listener at localhost/33753] zookeeper.ZKUtil(164): regionserver:40829-0x1004189558d0002, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 04:00:13,757 DEBUG [Listener at localhost/33753] zookeeper.ZKUtil(164): regionserver:40829-0x1004189558d0002, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-11 04:00:13,758 DEBUG [Listener at localhost/33753] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40829 2023-02-11 04:00:13,758 DEBUG [Listener at localhost/33753] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=40829 2023-02-11 04:00:13,758 DEBUG [Listener at localhost/33753] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=40829 2023-02-11 04:00:13,759 DEBUG [Listener at localhost/33753] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40829 2023-02-11 04:00:13,760 DEBUG [Listener at localhost/33753] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=40829 2023-02-11 04:00:13,772 INFO [Listener at localhost/33753] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=18 2023-02-11 04:00:13,772 INFO [Listener at localhost/33753] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:13,772 INFO [Listener at localhost/33753] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:13,772 INFO [Listener at localhost/33753] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-11 04:00:13,772 INFO [Listener at localhost/33753] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 04:00:13,773 INFO [Listener at localhost/33753] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-11 04:00:13,773 INFO [Listener at localhost/33753] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-11 04:00:13,774 INFO [Listener at localhost/33753] ipc.NettyRpcServer(120): Bind to /172.31.14.131:42385 2023-02-11 04:00:13,774 INFO [Listener at localhost/33753] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-11 04:00:13,775 INFO [Listener at localhost/33753] mob.MobFileCache(124): MobFileCache disabled 2023-02-11 04:00:13,776 INFO [Listener at localhost/33753] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 04:00:13,778 INFO [Listener at localhost/33753] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 04:00:13,779 INFO [Listener at localhost/33753] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:42385 connecting to ZooKeeper ensemble=127.0.0.1:63425 2023-02-11 04:00:13,784 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): regionserver:423850x0, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-11 04:00:13,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:42385-0x1004189558d0003 connected 2023-02-11 04:00:13,785 DEBUG [Listener at localhost/33753] zookeeper.ZKUtil(164): regionserver:42385-0x1004189558d0003, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 04:00:13,785 DEBUG [Listener at localhost/33753] zookeeper.ZKUtil(164): regionserver:42385-0x1004189558d0003, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 04:00:13,786 DEBUG [Listener at localhost/33753] zookeeper.ZKUtil(164): regionserver:42385-0x1004189558d0003, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-11 04:00:13,786 DEBUG [Listener at localhost/33753] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=42385 2023-02-11 04:00:13,787 DEBUG [Listener at localhost/33753] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=42385 2023-02-11 04:00:13,787 DEBUG [Listener at localhost/33753] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=42385 2023-02-11 04:00:13,788 DEBUG [Listener at localhost/33753] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=42385 2023-02-11 04:00:13,788 DEBUG [Listener at localhost/33753] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=42385 2023-02-11 04:00:13,789 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,41713,1676088012566 2023-02-11 04:00:13,798 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-11 04:00:13,799 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,41713,1676088012566 2023-02-11 04:00:13,815 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-11 04:00:13,816 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): regionserver:42385-0x1004189558d0003, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-11 04:00:13,815 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): regionserver:40829-0x1004189558d0002, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-11 04:00:13,816 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): regionserver:46851-0x1004189558d0001, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-11 04:00:13,816 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 04:00:13,817 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-11 04:00:13,819 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,41713,1676088012566 from backup master directory 2023-02-11 04:00:13,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-11 04:00:13,822 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,41713,1676088012566 2023-02-11 04:00:13,823 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-11 04:00:13,823 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-11 04:00:13,823 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase4.apache.org,41713,1676088012566 2023-02-11 04:00:13,826 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-02-11 04:00:13,827 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-02-11 04:00:13,916 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(628): Created cluster ID file at hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/hbase.id with ID: aba4dffd-e8bc-4776-92cf-c039ea23ea41 2023-02-11 04:00:13,960 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 04:00:13,976 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 04:00:14,026 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x1fbe9909 to 127.0.0.1:63425 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 04:00:14,058 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5060561b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 04:00:14,081 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-11 04:00:14,083 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-11 04:00:14,102 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-02-11 04:00:14,103 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-02-11 04:00:14,104 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-11 04:00:14,108 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-11 04:00:14,110 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 04:00:14,141 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7689): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/MasterData/data/master/store-tmp 2023-02-11 04:00:14,206 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:14,207 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-11 04:00:14,207 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 04:00:14,207 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 04:00:14,207 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-11 04:00:14,207 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 04:00:14,207 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 04:00:14,208 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676088014207Disabling compacts and flushes for region at 1676088014207Disabling writes for close at 1676088014207Writing region close event to WAL at 1676088014207Closed at 1676088014207 2023-02-11 04:00:14,209 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/MasterData/WALs/jenkins-hbase4.apache.org,41713,1676088012566 2023-02-11 04:00:14,231 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C41713%2C1676088012566, suffix=, logDir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/MasterData/WALs/jenkins-hbase4.apache.org,41713,1676088012566, archiveDir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/MasterData/oldWALs, maxLogs=10 2023-02-11 04:00:14,286 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34349,DS-8f0e379d-716c-4016-9c93-5a6633025c44,DISK] 2023-02-11 04:00:14,286 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37613,DS-7f7ff339-c799-4e70-9519-662960207563,DISK] 2023-02-11 04:00:14,286 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34161,DS-e59e7235-d091-4888-a8ac-54b667a6e5b2,DISK] 2023-02-11 04:00:14,294 DEBUG [RS-EventLoopGroup-5-3] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-11 04:00:14,356 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/MasterData/WALs/jenkins-hbase4.apache.org,41713,1676088012566/jenkins-hbase4.apache.org%2C41713%2C1676088012566.1676088014241 2023-02-11 04:00:14,356 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37613,DS-7f7ff339-c799-4e70-9519-662960207563,DISK], DatanodeInfoWithStorage[127.0.0.1:34161,DS-e59e7235-d091-4888-a8ac-54b667a6e5b2,DISK], DatanodeInfoWithStorage[127.0.0.1:34349,DS-8f0e379d-716c-4016-9c93-5a6633025c44,DISK]] 2023-02-11 04:00:14,357 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-11 04:00:14,357 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:14,360 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-11 04:00:14,361 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-11 04:00:14,419 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-11 04:00:14,426 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-11 04:00:14,448 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-11 04:00:14,459 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:14,465 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-11 04:00:14,466 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-11 04:00:14,479 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-11 04:00:14,484 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 04:00:14,485 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61316932, jitterRate=-0.08630651235580444}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:14,485 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676088014388Initializing all the Stores at 1676088014390 (+2 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676088014391 (+1 ms)Cleaning up temporary data from old regions at 1676088014472 (+81 ms)Cleaning up detritus from prior splits at 1676088014473 (+1 ms)Region opened successfully at 1676088014485 (+12 ms) 2023-02-11 04:00:14,486 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-11 04:00:14,505 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-11 04:00:14,506 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-11 04:00:14,508 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-11 04:00:14,509 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-02-11 04:00:14,540 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 30 msec 2023-02-11 04:00:14,540 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-11 04:00:14,563 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-02-11 04:00:14,568 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-02-11 04:00:14,591 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-11 04:00:14,594 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-11 04:00:14,595 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-02-11 04:00:14,600 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-11 04:00:14,604 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-11 04:00:14,606 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 04:00:14,608 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-11 04:00:14,608 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-11 04:00:14,620 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-11 04:00:14,624 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-11 04:00:14,624 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): regionserver:46851-0x1004189558d0001, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-11 04:00:14,625 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 04:00:14,624 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): regionserver:42385-0x1004189558d0003, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-11 04:00:14,624 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): regionserver:40829-0x1004189558d0002, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-11 04:00:14,627 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase4.apache.org,41713,1676088012566, sessionid=0x1004189558d0000, setting cluster-up flag (Was=false) 2023-02-11 04:00:14,640 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 04:00:14,646 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-11 04:00:14,648 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,41713,1676088012566 2023-02-11 04:00:14,653 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 04:00:14,658 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-11 04:00:14,660 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,41713,1676088012566 2023-02-11 04:00:14,662 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/.tmp 2023-02-11 04:00:14,691 INFO [RS:2;jenkins-hbase4:42385] regionserver.HRegionServer(952): ClusterId : aba4dffd-e8bc-4776-92cf-c039ea23ea41 2023-02-11 04:00:14,691 INFO [RS:1;jenkins-hbase4:40829] regionserver.HRegionServer(952): ClusterId : aba4dffd-e8bc-4776-92cf-c039ea23ea41 2023-02-11 04:00:14,691 INFO [RS:0;jenkins-hbase4:46851] regionserver.HRegionServer(952): ClusterId : aba4dffd-e8bc-4776-92cf-c039ea23ea41 2023-02-11 04:00:14,697 DEBUG [RS:0;jenkins-hbase4:46851] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-11 04:00:14,697 DEBUG [RS:1;jenkins-hbase4:40829] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-11 04:00:14,697 DEBUG [RS:2;jenkins-hbase4:42385] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-11 04:00:14,705 DEBUG [RS:0;jenkins-hbase4:46851] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-11 04:00:14,705 DEBUG [RS:2;jenkins-hbase4:42385] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-11 04:00:14,705 DEBUG [RS:1;jenkins-hbase4:40829] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-11 04:00:14,705 DEBUG [RS:2;jenkins-hbase4:42385] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-11 04:00:14,705 DEBUG [RS:0;jenkins-hbase4:46851] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-11 04:00:14,706 DEBUG [RS:1;jenkins-hbase4:40829] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-11 04:00:14,710 DEBUG [RS:2;jenkins-hbase4:42385] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-11 04:00:14,710 DEBUG [RS:0;jenkins-hbase4:46851] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-11 04:00:14,710 DEBUG [RS:1;jenkins-hbase4:40829] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-11 04:00:14,712 DEBUG [RS:2;jenkins-hbase4:42385] zookeeper.ReadOnlyZKClient(139): Connect 0x139497cc to 127.0.0.1:63425 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 04:00:14,713 DEBUG [RS:0;jenkins-hbase4:46851] zookeeper.ReadOnlyZKClient(139): Connect 0x0047547e to 127.0.0.1:63425 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 04:00:14,713 DEBUG [RS:1;jenkins-hbase4:40829] zookeeper.ReadOnlyZKClient(139): Connect 0x748d25ce to 127.0.0.1:63425 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 04:00:14,726 DEBUG [RS:1;jenkins-hbase4:40829] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7a4ad52f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 04:00:14,726 DEBUG [RS:0;jenkins-hbase4:46851] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@e80aa06, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 04:00:14,726 DEBUG [RS:2;jenkins-hbase4:42385] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d537bce, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 04:00:14,727 DEBUG [RS:1;jenkins-hbase4:40829] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e754ebe, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-11 04:00:14,727 DEBUG [RS:0;jenkins-hbase4:46851] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@abf3f0e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-11 04:00:14,727 DEBUG [RS:2;jenkins-hbase4:42385] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@196cfcbc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-11 04:00:14,749 DEBUG [RS:0;jenkins-hbase4:46851] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:46851 2023-02-11 04:00:14,750 DEBUG [RS:1;jenkins-hbase4:40829] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:40829 2023-02-11 04:00:14,753 DEBUG [RS:2;jenkins-hbase4:42385] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:42385 2023-02-11 04:00:14,753 INFO [RS:0;jenkins-hbase4:46851] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-11 04:00:14,754 INFO [RS:0;jenkins-hbase4:46851] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-11 04:00:14,753 INFO [RS:2;jenkins-hbase4:42385] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-11 04:00:14,754 DEBUG [RS:0;jenkins-hbase4:46851] regionserver.HRegionServer(1023): About to register with Master. 2023-02-11 04:00:14,754 INFO [RS:1;jenkins-hbase4:40829] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-11 04:00:14,754 INFO [RS:2;jenkins-hbase4:42385] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-11 04:00:14,754 INFO [RS:1;jenkins-hbase4:40829] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-11 04:00:14,754 DEBUG [RS:1;jenkins-hbase4:40829] regionserver.HRegionServer(1023): About to register with Master. 2023-02-11 04:00:14,754 DEBUG [RS:2;jenkins-hbase4:42385] regionserver.HRegionServer(1023): About to register with Master. 2023-02-11 04:00:14,757 INFO [RS:2;jenkins-hbase4:42385] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,41713,1676088012566 with isa=jenkins-hbase4.apache.org/172.31.14.131:42385, startcode=1676088013771 2023-02-11 04:00:14,757 INFO [RS:1;jenkins-hbase4:40829] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,41713,1676088012566 with isa=jenkins-hbase4.apache.org/172.31.14.131:40829, startcode=1676088013739 2023-02-11 04:00:14,757 INFO [RS:0;jenkins-hbase4:46851] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,41713,1676088012566 with isa=jenkins-hbase4.apache.org/172.31.14.131:46851, startcode=1676088013694 2023-02-11 04:00:14,774 DEBUG [RS:0;jenkins-hbase4:46851] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-11 04:00:14,774 DEBUG [RS:1;jenkins-hbase4:40829] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-11 04:00:14,774 DEBUG [RS:2;jenkins-hbase4:42385] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-11 04:00:14,795 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-02-11 04:00:14,814 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-11 04:00:14,814 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-11 04:00:14,814 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-11 04:00:14,814 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-11 04:00:14,815 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-02-11 04:00:14,815 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:14,815 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-11 04:00:14,815 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:14,819 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676088044819 2023-02-11 04:00:14,822 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-11 04:00:14,826 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-02-11 04:00:14,827 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-02-11 04:00:14,832 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-11 04:00:14,834 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-11 04:00:14,835 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42851, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-02-11 04:00:14,835 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46705, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-02-11 04:00:14,835 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56365, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-02-11 04:00:14,843 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-11 04:00:14,843 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-11 04:00:14,843 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-11 04:00:14,843 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-11 04:00:14,844 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:14,845 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-11 04:00:14,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41713] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:14,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41713] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:14,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:14,847 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-11 04:00:14,848 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-11 04:00:14,851 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-11 04:00:14,851 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-11 04:00:14,853 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676088014852,5,FailOnTimeoutGroup] 2023-02-11 04:00:14,853 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676088014853,5,FailOnTimeoutGroup] 2023-02-11 04:00:14,853 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:14,853 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-11 04:00:14,854 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:14,855 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:14,880 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-11 04:00:14,881 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-11 04:00:14,881 INFO [PEWorker-1] regionserver.HRegion(7671): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6 2023-02-11 04:00:14,888 DEBUG [RS:0;jenkins-hbase4:46851] regionserver.HRegionServer(2829): Master is not running yet 2023-02-11 04:00:14,888 DEBUG [RS:2;jenkins-hbase4:42385] regionserver.HRegionServer(2829): Master is not running yet 2023-02-11 04:00:14,888 DEBUG [RS:1;jenkins-hbase4:40829] regionserver.HRegionServer(2829): Master is not running yet 2023-02-11 04:00:14,889 WARN [RS:2;jenkins-hbase4:42385] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-11 04:00:14,888 WARN [RS:0;jenkins-hbase4:46851] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-11 04:00:14,889 WARN [RS:1;jenkins-hbase4:40829] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-11 04:00:14,911 DEBUG [PEWorker-1] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:14,914 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-11 04:00:14,916 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/meta/1588230740/info 2023-02-11 04:00:14,916 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-11 04:00:14,917 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:14,918 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-11 04:00:14,920 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/meta/1588230740/rep_barrier 2023-02-11 04:00:14,921 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-11 04:00:14,921 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:14,921 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-11 04:00:14,923 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/meta/1588230740/table 2023-02-11 04:00:14,924 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-11 04:00:14,925 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:14,926 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/meta/1588230740 2023-02-11 04:00:14,927 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/meta/1588230740 2023-02-11 04:00:14,931 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-11 04:00:14,933 DEBUG [PEWorker-1] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-11 04:00:14,936 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 04:00:14,937 INFO [PEWorker-1] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65826023, jitterRate=-0.01911582052707672}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-11 04:00:14,937 DEBUG [PEWorker-1] regionserver.HRegion(964): Region open journal for 1588230740: Writing region info on filesystem at 1676088014912Initializing all the Stores at 1676088014913 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676088014913Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676088014913Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676088014913Cleaning up temporary data from old regions at 1676088014929 (+16 ms)Cleaning up detritus from prior splits at 1676088014929Region opened successfully at 1676088014937 (+8 ms) 2023-02-11 04:00:14,937 DEBUG [PEWorker-1] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-11 04:00:14,937 INFO [PEWorker-1] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-11 04:00:14,937 DEBUG [PEWorker-1] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-11 04:00:14,937 DEBUG [PEWorker-1] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-11 04:00:14,937 DEBUG [PEWorker-1] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-11 04:00:14,938 INFO [PEWorker-1] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-11 04:00:14,938 DEBUG [PEWorker-1] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676088014937Disabling compacts and flushes for region at 1676088014937Disabling writes for close at 1676088014937Writing region close event to WAL at 1676088014938 (+1 ms)Closed at 1676088014938 2023-02-11 04:00:14,943 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-02-11 04:00:14,943 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-02-11 04:00:14,951 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-11 04:00:14,962 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-11 04:00:14,966 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-02-11 04:00:14,990 INFO [RS:2;jenkins-hbase4:42385] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,41713,1676088012566 with isa=jenkins-hbase4.apache.org/172.31.14.131:42385, startcode=1676088013771 2023-02-11 04:00:14,990 INFO [RS:0;jenkins-hbase4:46851] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,41713,1676088012566 with isa=jenkins-hbase4.apache.org/172.31.14.131:46851, startcode=1676088013694 2023-02-11 04:00:14,990 INFO [RS:1;jenkins-hbase4:40829] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,41713,1676088012566 with isa=jenkins-hbase4.apache.org/172.31.14.131:40829, startcode=1676088013739 2023-02-11 04:00:14,996 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41713] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:14,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41713] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:14,997 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:15,002 DEBUG [RS:2;jenkins-hbase4:42385] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6 2023-02-11 04:00:15,002 DEBUG [RS:0;jenkins-hbase4:46851] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6 2023-02-11 04:00:15,002 DEBUG [RS:1;jenkins-hbase4:40829] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6 2023-02-11 04:00:15,003 DEBUG [RS:0;jenkins-hbase4:46851] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:37271 2023-02-11 04:00:15,002 DEBUG [RS:2;jenkins-hbase4:42385] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:37271 2023-02-11 04:00:15,003 DEBUG [RS:2;jenkins-hbase4:42385] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-11 04:00:15,003 DEBUG [RS:0;jenkins-hbase4:46851] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-11 04:00:15,003 DEBUG [RS:1;jenkins-hbase4:40829] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:37271 2023-02-11 04:00:15,003 DEBUG [RS:1;jenkins-hbase4:40829] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-11 04:00:15,014 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 04:00:15,014 DEBUG [RS:0;jenkins-hbase4:46851] zookeeper.ZKUtil(162): regionserver:46851-0x1004189558d0001, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:15,014 DEBUG [RS:1;jenkins-hbase4:40829] zookeeper.ZKUtil(162): regionserver:40829-0x1004189558d0002, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:15,014 DEBUG [RS:2;jenkins-hbase4:42385] zookeeper.ZKUtil(162): regionserver:42385-0x1004189558d0003, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:15,014 WARN [RS:1;jenkins-hbase4:40829] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-11 04:00:15,014 WARN [RS:0;jenkins-hbase4:46851] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-11 04:00:15,015 INFO [RS:1;jenkins-hbase4:40829] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 04:00:15,015 INFO [RS:0;jenkins-hbase4:46851] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 04:00:15,015 WARN [RS:2;jenkins-hbase4:42385] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-11 04:00:15,016 DEBUG [RS:0;jenkins-hbase4:46851] regionserver.HRegionServer(1947): logDir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/WALs/jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:15,016 INFO [RS:2;jenkins-hbase4:42385] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 04:00:15,016 DEBUG [RS:1;jenkins-hbase4:40829] regionserver.HRegionServer(1947): logDir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/WALs/jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:15,016 DEBUG [RS:2;jenkins-hbase4:42385] regionserver.HRegionServer(1947): logDir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/WALs/jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:15,016 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,46851,1676088013694] 2023-02-11 04:00:15,016 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,40829,1676088013739] 2023-02-11 04:00:15,016 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,42385,1676088013771] 2023-02-11 04:00:15,029 DEBUG [RS:2;jenkins-hbase4:42385] zookeeper.ZKUtil(162): regionserver:42385-0x1004189558d0003, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:15,029 DEBUG [RS:1;jenkins-hbase4:40829] zookeeper.ZKUtil(162): regionserver:40829-0x1004189558d0002, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:15,029 DEBUG [RS:0;jenkins-hbase4:46851] zookeeper.ZKUtil(162): regionserver:46851-0x1004189558d0001, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:15,029 DEBUG [RS:2;jenkins-hbase4:42385] zookeeper.ZKUtil(162): regionserver:42385-0x1004189558d0003, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:15,029 DEBUG [RS:1;jenkins-hbase4:40829] zookeeper.ZKUtil(162): regionserver:40829-0x1004189558d0002, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:15,030 DEBUG [RS:0;jenkins-hbase4:46851] zookeeper.ZKUtil(162): regionserver:46851-0x1004189558d0001, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:15,030 DEBUG [RS:2;jenkins-hbase4:42385] zookeeper.ZKUtil(162): regionserver:42385-0x1004189558d0003, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:15,030 DEBUG [RS:1;jenkins-hbase4:40829] zookeeper.ZKUtil(162): regionserver:40829-0x1004189558d0002, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:15,030 DEBUG [RS:0;jenkins-hbase4:46851] zookeeper.ZKUtil(162): regionserver:46851-0x1004189558d0001, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:15,041 DEBUG [RS:0;jenkins-hbase4:46851] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-11 04:00:15,041 DEBUG [RS:1;jenkins-hbase4:40829] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-11 04:00:15,041 DEBUG [RS:2;jenkins-hbase4:42385] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-11 04:00:15,050 INFO [RS:2;jenkins-hbase4:42385] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-11 04:00:15,050 INFO [RS:1;jenkins-hbase4:40829] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-11 04:00:15,050 INFO [RS:0;jenkins-hbase4:46851] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-11 04:00:15,070 INFO [RS:1;jenkins-hbase4:40829] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-11 04:00:15,070 INFO [RS:0;jenkins-hbase4:46851] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-11 04:00:15,073 INFO [RS:2;jenkins-hbase4:42385] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-11 04:00:15,075 INFO [RS:0;jenkins-hbase4:46851] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-11 04:00:15,075 INFO [RS:1;jenkins-hbase4:40829] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-11 04:00:15,076 INFO [RS:0;jenkins-hbase4:46851] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:15,075 INFO [RS:2;jenkins-hbase4:42385] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-11 04:00:15,076 INFO [RS:1;jenkins-hbase4:40829] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:15,076 INFO [RS:2;jenkins-hbase4:42385] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:15,077 INFO [RS:0;jenkins-hbase4:46851] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-11 04:00:15,077 INFO [RS:1;jenkins-hbase4:40829] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-11 04:00:15,077 INFO [RS:2;jenkins-hbase4:42385] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-11 04:00:15,085 INFO [RS:0;jenkins-hbase4:46851] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:15,085 INFO [RS:1;jenkins-hbase4:40829] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:15,085 INFO [RS:2;jenkins-hbase4:42385] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:15,085 DEBUG [RS:0;jenkins-hbase4:46851] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:15,085 DEBUG [RS:2;jenkins-hbase4:42385] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:15,085 DEBUG [RS:1;jenkins-hbase4:40829] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:15,085 DEBUG [RS:2;jenkins-hbase4:42385] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:15,085 DEBUG [RS:1;jenkins-hbase4:40829] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:15,085 DEBUG [RS:2;jenkins-hbase4:42385] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:15,085 DEBUG [RS:0;jenkins-hbase4:46851] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:15,086 DEBUG [RS:2;jenkins-hbase4:42385] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:15,086 DEBUG [RS:1;jenkins-hbase4:40829] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:15,086 DEBUG [RS:2;jenkins-hbase4:42385] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:15,086 DEBUG [RS:1;jenkins-hbase4:40829] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:15,086 DEBUG [RS:0;jenkins-hbase4:46851] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:15,086 DEBUG [RS:1;jenkins-hbase4:40829] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:15,086 DEBUG [RS:2;jenkins-hbase4:42385] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-11 04:00:15,086 DEBUG [RS:1;jenkins-hbase4:40829] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-11 04:00:15,086 DEBUG [RS:2;jenkins-hbase4:42385] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:15,086 DEBUG [RS:0;jenkins-hbase4:46851] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:15,086 DEBUG [RS:2;jenkins-hbase4:42385] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:15,086 DEBUG [RS:1;jenkins-hbase4:40829] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:15,086 DEBUG [RS:2;jenkins-hbase4:42385] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:15,086 DEBUG [RS:0;jenkins-hbase4:46851] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:15,087 DEBUG [RS:2;jenkins-hbase4:42385] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:15,087 DEBUG [RS:1;jenkins-hbase4:40829] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:15,087 DEBUG [RS:0;jenkins-hbase4:46851] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-11 04:00:15,087 DEBUG [RS:1;jenkins-hbase4:40829] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:15,087 DEBUG [RS:0;jenkins-hbase4:46851] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:15,087 DEBUG [RS:1;jenkins-hbase4:40829] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:15,087 DEBUG [RS:0;jenkins-hbase4:46851] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:15,088 DEBUG [RS:0;jenkins-hbase4:46851] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:15,088 INFO [RS:2;jenkins-hbase4:42385] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:15,088 DEBUG [RS:0;jenkins-hbase4:46851] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 04:00:15,088 INFO [RS:2;jenkins-hbase4:42385] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:15,088 INFO [RS:2;jenkins-hbase4:42385] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:15,092 INFO [RS:1;jenkins-hbase4:40829] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:15,092 INFO [RS:1;jenkins-hbase4:40829] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:15,092 INFO [RS:1;jenkins-hbase4:40829] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:15,096 INFO [RS:0;jenkins-hbase4:46851] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:15,096 INFO [RS:0;jenkins-hbase4:46851] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:15,096 INFO [RS:0;jenkins-hbase4:46851] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:15,108 INFO [RS:2;jenkins-hbase4:42385] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-11 04:00:15,108 INFO [RS:0;jenkins-hbase4:46851] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-11 04:00:15,108 INFO [RS:1;jenkins-hbase4:40829] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-11 04:00:15,111 INFO [RS:1;jenkins-hbase4:40829] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40829,1676088013739-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:15,111 INFO [RS:2;jenkins-hbase4:42385] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42385,1676088013771-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:15,111 INFO [RS:0;jenkins-hbase4:46851] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46851,1676088013694-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:15,118 DEBUG [jenkins-hbase4:41713] assignment.AssignmentManager(2178): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-02-11 04:00:15,123 DEBUG [jenkins-hbase4:41713] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 04:00:15,129 DEBUG [jenkins-hbase4:41713] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 04:00:15,130 DEBUG [jenkins-hbase4:41713] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 04:00:15,130 DEBUG [jenkins-hbase4:41713] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 04:00:15,130 DEBUG [jenkins-hbase4:41713] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 04:00:15,130 INFO [RS:1;jenkins-hbase4:40829] regionserver.Replication(203): jenkins-hbase4.apache.org,40829,1676088013739 started 2023-02-11 04:00:15,130 INFO [RS:1;jenkins-hbase4:40829] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,40829,1676088013739, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:40829, sessionid=0x1004189558d0002 2023-02-11 04:00:15,131 DEBUG [RS:1;jenkins-hbase4:40829] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-11 04:00:15,131 DEBUG [RS:1;jenkins-hbase4:40829] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:15,131 DEBUG [RS:1;jenkins-hbase4:40829] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,40829,1676088013739' 2023-02-11 04:00:15,131 DEBUG [RS:1;jenkins-hbase4:40829] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-11 04:00:15,132 DEBUG [RS:1;jenkins-hbase4:40829] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-11 04:00:15,132 DEBUG [RS:1;jenkins-hbase4:40829] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-11 04:00:15,132 DEBUG [RS:1;jenkins-hbase4:40829] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-11 04:00:15,133 DEBUG [RS:1;jenkins-hbase4:40829] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:15,133 DEBUG [RS:1;jenkins-hbase4:40829] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,40829,1676088013739' 2023-02-11 04:00:15,133 DEBUG [RS:1;jenkins-hbase4:40829] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-11 04:00:15,133 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,40829,1676088013739, state=OPENING 2023-02-11 04:00:15,133 DEBUG [RS:1;jenkins-hbase4:40829] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-11 04:00:15,134 DEBUG [RS:1;jenkins-hbase4:40829] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-11 04:00:15,134 INFO [RS:1;jenkins-hbase4:40829] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-11 04:00:15,134 INFO [RS:1;jenkins-hbase4:40829] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-11 04:00:15,136 INFO [RS:2;jenkins-hbase4:42385] regionserver.Replication(203): jenkins-hbase4.apache.org,42385,1676088013771 started 2023-02-11 04:00:15,136 INFO [RS:2;jenkins-hbase4:42385] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,42385,1676088013771, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:42385, sessionid=0x1004189558d0003 2023-02-11 04:00:15,138 DEBUG [RS:2;jenkins-hbase4:42385] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-11 04:00:15,138 INFO [RS:0;jenkins-hbase4:46851] regionserver.Replication(203): jenkins-hbase4.apache.org,46851,1676088013694 started 2023-02-11 04:00:15,138 DEBUG [RS:2;jenkins-hbase4:42385] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:15,139 INFO [RS:0;jenkins-hbase4:46851] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,46851,1676088013694, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:46851, sessionid=0x1004189558d0001 2023-02-11 04:00:15,139 DEBUG [RS:2;jenkins-hbase4:42385] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42385,1676088013771' 2023-02-11 04:00:15,140 DEBUG [RS:2;jenkins-hbase4:42385] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-11 04:00:15,140 DEBUG [RS:0;jenkins-hbase4:46851] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-11 04:00:15,140 DEBUG [RS:0;jenkins-hbase4:46851] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:15,140 DEBUG [RS:0;jenkins-hbase4:46851] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,46851,1676088013694' 2023-02-11 04:00:15,140 DEBUG [RS:0;jenkins-hbase4:46851] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-11 04:00:15,141 DEBUG [RS:2;jenkins-hbase4:42385] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-11 04:00:15,141 DEBUG [RS:0;jenkins-hbase4:46851] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-11 04:00:15,141 DEBUG [RS:2;jenkins-hbase4:42385] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-11 04:00:15,141 DEBUG [RS:2;jenkins-hbase4:42385] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-11 04:00:15,141 DEBUG [RS:0;jenkins-hbase4:46851] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-11 04:00:15,141 DEBUG [RS:2;jenkins-hbase4:42385] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:15,142 DEBUG [RS:2;jenkins-hbase4:42385] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42385,1676088013771' 2023-02-11 04:00:15,142 DEBUG [RS:2;jenkins-hbase4:42385] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-11 04:00:15,142 DEBUG [RS:0;jenkins-hbase4:46851] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-11 04:00:15,142 DEBUG [RS:0;jenkins-hbase4:46851] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:15,142 DEBUG [RS:0;jenkins-hbase4:46851] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,46851,1676088013694' 2023-02-11 04:00:15,142 DEBUG [RS:0;jenkins-hbase4:46851] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-11 04:00:15,142 DEBUG [RS:2;jenkins-hbase4:42385] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-11 04:00:15,143 DEBUG [RS:0;jenkins-hbase4:46851] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-11 04:00:15,144 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-02-11 04:00:15,144 DEBUG [RS:2;jenkins-hbase4:42385] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-11 04:00:15,144 DEBUG [RS:0;jenkins-hbase4:46851] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-11 04:00:15,144 INFO [RS:0;jenkins-hbase4:46851] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-11 04:00:15,144 INFO [RS:2;jenkins-hbase4:42385] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-11 04:00:15,144 INFO [RS:0;jenkins-hbase4:46851] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-11 04:00:15,145 INFO [RS:2;jenkins-hbase4:42385] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-11 04:00:15,146 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 04:00:15,146 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-11 04:00:15,151 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:15,244 INFO [RS:1;jenkins-hbase4:40829] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C40829%2C1676088013739, suffix=, logDir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/WALs/jenkins-hbase4.apache.org,40829,1676088013739, archiveDir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/oldWALs, maxLogs=32 2023-02-11 04:00:15,248 INFO [RS:0;jenkins-hbase4:46851] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46851%2C1676088013694, suffix=, logDir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/WALs/jenkins-hbase4.apache.org,46851,1676088013694, archiveDir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/oldWALs, maxLogs=32 2023-02-11 04:00:15,248 INFO [RS:2;jenkins-hbase4:42385] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42385%2C1676088013771, suffix=, logDir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/WALs/jenkins-hbase4.apache.org,42385,1676088013771, archiveDir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/oldWALs, maxLogs=32 2023-02-11 04:00:15,265 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34161,DS-e59e7235-d091-4888-a8ac-54b667a6e5b2,DISK] 2023-02-11 04:00:15,265 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37613,DS-7f7ff339-c799-4e70-9519-662960207563,DISK] 2023-02-11 04:00:15,265 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34349,DS-8f0e379d-716c-4016-9c93-5a6633025c44,DISK] 2023-02-11 04:00:15,272 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37613,DS-7f7ff339-c799-4e70-9519-662960207563,DISK] 2023-02-11 04:00:15,273 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34161,DS-e59e7235-d091-4888-a8ac-54b667a6e5b2,DISK] 2023-02-11 04:00:15,273 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34349,DS-8f0e379d-716c-4016-9c93-5a6633025c44,DISK] 2023-02-11 04:00:15,280 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37613,DS-7f7ff339-c799-4e70-9519-662960207563,DISK] 2023-02-11 04:00:15,280 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34161,DS-e59e7235-d091-4888-a8ac-54b667a6e5b2,DISK] 2023-02-11 04:00:15,280 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34349,DS-8f0e379d-716c-4016-9c93-5a6633025c44,DISK] 2023-02-11 04:00:15,287 INFO [RS:1;jenkins-hbase4:40829] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/WALs/jenkins-hbase4.apache.org,40829,1676088013739/jenkins-hbase4.apache.org%2C40829%2C1676088013739.1676088015246 2023-02-11 04:00:15,287 DEBUG [RS:1;jenkins-hbase4:40829] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34349,DS-8f0e379d-716c-4016-9c93-5a6633025c44,DISK], DatanodeInfoWithStorage[127.0.0.1:34161,DS-e59e7235-d091-4888-a8ac-54b667a6e5b2,DISK], DatanodeInfoWithStorage[127.0.0.1:37613,DS-7f7ff339-c799-4e70-9519-662960207563,DISK]] 2023-02-11 04:00:15,288 INFO [RS:0;jenkins-hbase4:46851] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/WALs/jenkins-hbase4.apache.org,46851,1676088013694/jenkins-hbase4.apache.org%2C46851%2C1676088013694.1676088015249 2023-02-11 04:00:15,289 DEBUG [RS:0;jenkins-hbase4:46851] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37613,DS-7f7ff339-c799-4e70-9519-662960207563,DISK], DatanodeInfoWithStorage[127.0.0.1:34349,DS-8f0e379d-716c-4016-9c93-5a6633025c44,DISK], DatanodeInfoWithStorage[127.0.0.1:34161,DS-e59e7235-d091-4888-a8ac-54b667a6e5b2,DISK]] 2023-02-11 04:00:15,289 INFO [RS:2;jenkins-hbase4:42385] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/WALs/jenkins-hbase4.apache.org,42385,1676088013771/jenkins-hbase4.apache.org%2C42385%2C1676088013771.1676088015249 2023-02-11 04:00:15,290 DEBUG [RS:2;jenkins-hbase4:42385] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34161,DS-e59e7235-d091-4888-a8ac-54b667a6e5b2,DISK], DatanodeInfoWithStorage[127.0.0.1:34349,DS-8f0e379d-716c-4016-9c93-5a6633025c44,DISK], DatanodeInfoWithStorage[127.0.0.1:37613,DS-7f7ff339-c799-4e70-9519-662960207563,DISK]] 2023-02-11 04:00:15,339 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:15,341 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-11 04:00:15,344 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55106, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-11 04:00:15,356 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-11 04:00:15,357 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 04:00:15,360 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C40829%2C1676088013739.meta, suffix=.meta, logDir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/WALs/jenkins-hbase4.apache.org,40829,1676088013739, archiveDir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/oldWALs, maxLogs=32 2023-02-11 04:00:15,379 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34161,DS-e59e7235-d091-4888-a8ac-54b667a6e5b2,DISK] 2023-02-11 04:00:15,381 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34349,DS-8f0e379d-716c-4016-9c93-5a6633025c44,DISK] 2023-02-11 04:00:15,381 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37613,DS-7f7ff339-c799-4e70-9519-662960207563,DISK] 2023-02-11 04:00:15,384 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/WALs/jenkins-hbase4.apache.org,40829,1676088013739/jenkins-hbase4.apache.org%2C40829%2C1676088013739.meta.1676088015361.meta 2023-02-11 04:00:15,385 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34161,DS-e59e7235-d091-4888-a8ac-54b667a6e5b2,DISK], DatanodeInfoWithStorage[127.0.0.1:34349,DS-8f0e379d-716c-4016-9c93-5a6633025c44,DISK], DatanodeInfoWithStorage[127.0.0.1:37613,DS-7f7ff339-c799-4e70-9519-662960207563,DISK]] 2023-02-11 04:00:15,385 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-11 04:00:15,387 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-11 04:00:15,403 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-11 04:00:15,408 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-11 04:00:15,417 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-11 04:00:15,417 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:15,417 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-11 04:00:15,417 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-11 04:00:15,420 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-11 04:00:15,422 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/meta/1588230740/info 2023-02-11 04:00:15,422 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/meta/1588230740/info 2023-02-11 04:00:15,423 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-11 04:00:15,423 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:15,424 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-11 04:00:15,425 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/meta/1588230740/rep_barrier 2023-02-11 04:00:15,425 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/meta/1588230740/rep_barrier 2023-02-11 04:00:15,426 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-11 04:00:15,427 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:15,427 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-11 04:00:15,429 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/meta/1588230740/table 2023-02-11 04:00:15,429 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/meta/1588230740/table 2023-02-11 04:00:15,430 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-11 04:00:15,431 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:15,432 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/meta/1588230740 2023-02-11 04:00:15,435 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/meta/1588230740 2023-02-11 04:00:15,439 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-11 04:00:15,441 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-11 04:00:15,442 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68086704, jitterRate=0.014570951461791992}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-11 04:00:15,443 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1676088015417Writing region info on filesystem at 1676088015418 (+1 ms)Initializing all the Stores at 1676088015419 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676088015420 (+1 ms)Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676088015420Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676088015420Cleaning up temporary data from old regions at 1676088015437 (+17 ms)Cleaning up detritus from prior splits at 1676088015438 (+1 ms)Running coprocessor post-open hooks at 1676088015443 (+5 ms)Region opened successfully at 1676088015443 2023-02-11 04:00:15,454 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1676088015331 2023-02-11 04:00:15,471 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-11 04:00:15,472 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-11 04:00:15,472 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,40829,1676088013739, state=OPEN 2023-02-11 04:00:15,475 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-11 04:00:15,475 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-11 04:00:15,479 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-02-11 04:00:15,479 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,40829,1676088013739 in 325 msec 2023-02-11 04:00:15,483 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-02-11 04:00:15,483 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 528 msec 2023-02-11 04:00:15,488 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 779 msec 2023-02-11 04:00:15,489 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676088013812, completionTime=-1 2023-02-11 04:00:15,489 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-11 04:00:15,489 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-11 04:00:15,553 DEBUG [hconnection-0x48801dbb-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 04:00:15,556 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55116, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 04:00:15,571 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-11 04:00:15,572 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676088075571 2023-02-11 04:00:15,572 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676088135572 2023-02-11 04:00:15,572 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 82 msec 2023-02-11 04:00:15,594 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,41713,1676088012566-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:15,594 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,41713,1676088012566-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:15,594 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,41713,1676088012566-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:15,596 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:41713, period=300000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:15,596 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-11 04:00:15,605 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-02-11 04:00:15,613 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-02-11 04:00:15,614 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2138): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-11 04:00:15,625 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-02-11 04:00:15,627 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-02-11 04:00:15,629 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-11 04:00:15,652 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/hbase/namespace/e892275aa04716c37d873e5baba493e7 2023-02-11 04:00:15,655 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/hbase/namespace/e892275aa04716c37d873e5baba493e7 empty. 2023-02-11 04:00:15,655 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/hbase/namespace/e892275aa04716c37d873e5baba493e7 2023-02-11 04:00:15,655 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-02-11 04:00:15,692 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-02-11 04:00:15,694 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7671): creating {ENCODED => e892275aa04716c37d873e5baba493e7, NAME => 'hbase:namespace,,1676088015613.e892275aa04716c37d873e5baba493e7.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp 2023-02-11 04:00:15,712 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676088015613.e892275aa04716c37d873e5baba493e7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:15,712 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1603): Closing e892275aa04716c37d873e5baba493e7, disabling compactions & flushes 2023-02-11 04:00:15,712 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676088015613.e892275aa04716c37d873e5baba493e7. 2023-02-11 04:00:15,712 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676088015613.e892275aa04716c37d873e5baba493e7. 2023-02-11 04:00:15,713 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676088015613.e892275aa04716c37d873e5baba493e7. after waiting 0 ms 2023-02-11 04:00:15,713 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676088015613.e892275aa04716c37d873e5baba493e7. 2023-02-11 04:00:15,713 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676088015613.e892275aa04716c37d873e5baba493e7. 2023-02-11 04:00:15,713 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1557): Region close journal for e892275aa04716c37d873e5baba493e7: Waiting for close lock at 1676088015712Disabling compacts and flushes for region at 1676088015712Disabling writes for close at 1676088015713 (+1 ms)Writing region close event to WAL at 1676088015713Closed at 1676088015713 2023-02-11 04:00:15,717 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-02-11 04:00:15,733 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1676088015613.e892275aa04716c37d873e5baba493e7.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676088015721"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088015721"}]},"ts":"1676088015721"} 2023-02-11 04:00:15,760 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-11 04:00:15,762 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-11 04:00:15,766 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088015762"}]},"ts":"1676088015762"} 2023-02-11 04:00:15,770 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-02-11 04:00:15,774 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 04:00:15,775 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 04:00:15,775 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 04:00:15,775 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 04:00:15,775 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 04:00:15,778 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=e892275aa04716c37d873e5baba493e7, ASSIGN}] 2023-02-11 04:00:15,780 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=e892275aa04716c37d873e5baba493e7, ASSIGN 2023-02-11 04:00:15,782 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=e892275aa04716c37d873e5baba493e7, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40829,1676088013739; forceNewPlan=false, retain=false 2023-02-11 04:00:15,934 INFO [jenkins-hbase4:41713] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-11 04:00:15,935 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=e892275aa04716c37d873e5baba493e7, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:15,935 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1676088015613.e892275aa04716c37d873e5baba493e7.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676088015935"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088015935"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088015935"}]},"ts":"1676088015935"} 2023-02-11 04:00:15,939 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure e892275aa04716c37d873e5baba493e7, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:16,102 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1676088015613.e892275aa04716c37d873e5baba493e7. 2023-02-11 04:00:16,103 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e892275aa04716c37d873e5baba493e7, NAME => 'hbase:namespace,,1676088015613.e892275aa04716c37d873e5baba493e7.', STARTKEY => '', ENDKEY => ''} 2023-02-11 04:00:16,103 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace e892275aa04716c37d873e5baba493e7 2023-02-11 04:00:16,104 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676088015613.e892275aa04716c37d873e5baba493e7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:16,104 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e892275aa04716c37d873e5baba493e7 2023-02-11 04:00:16,104 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e892275aa04716c37d873e5baba493e7 2023-02-11 04:00:16,106 INFO [StoreOpener-e892275aa04716c37d873e5baba493e7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region e892275aa04716c37d873e5baba493e7 2023-02-11 04:00:16,108 DEBUG [StoreOpener-e892275aa04716c37d873e5baba493e7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/namespace/e892275aa04716c37d873e5baba493e7/info 2023-02-11 04:00:16,108 DEBUG [StoreOpener-e892275aa04716c37d873e5baba493e7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/namespace/e892275aa04716c37d873e5baba493e7/info 2023-02-11 04:00:16,108 INFO [StoreOpener-e892275aa04716c37d873e5baba493e7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e892275aa04716c37d873e5baba493e7 columnFamilyName info 2023-02-11 04:00:16,109 INFO [StoreOpener-e892275aa04716c37d873e5baba493e7-1] regionserver.HStore(310): Store=e892275aa04716c37d873e5baba493e7/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:16,111 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/namespace/e892275aa04716c37d873e5baba493e7 2023-02-11 04:00:16,111 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/namespace/e892275aa04716c37d873e5baba493e7 2023-02-11 04:00:16,115 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e892275aa04716c37d873e5baba493e7 2023-02-11 04:00:16,119 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/namespace/e892275aa04716c37d873e5baba493e7/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 04:00:16,119 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e892275aa04716c37d873e5baba493e7; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71182197, jitterRate=0.060697391629219055}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:16,120 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e892275aa04716c37d873e5baba493e7: Running coprocessor pre-open hook at 1676088016104Writing region info on filesystem at 1676088016104Initializing all the Stores at 1676088016105 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676088016105Cleaning up temporary data from old regions at 1676088016112 (+7 ms)Cleaning up detritus from prior splits at 1676088016112Running coprocessor post-open hooks at 1676088016119 (+7 ms)Region opened successfully at 1676088016119 2023-02-11 04:00:16,121 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1676088015613.e892275aa04716c37d873e5baba493e7., pid=6, masterSystemTime=1676088016093 2023-02-11 04:00:16,125 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1676088015613.e892275aa04716c37d873e5baba493e7. 2023-02-11 04:00:16,125 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1676088015613.e892275aa04716c37d873e5baba493e7. 2023-02-11 04:00:16,126 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=e892275aa04716c37d873e5baba493e7, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:16,127 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1676088015613.e892275aa04716c37d873e5baba493e7.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676088016126"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088016126"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088016126"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088016126"}]},"ts":"1676088016126"} 2023-02-11 04:00:16,136 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-02-11 04:00:16,136 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure e892275aa04716c37d873e5baba493e7, server=jenkins-hbase4.apache.org,40829,1676088013739 in 193 msec 2023-02-11 04:00:16,140 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-02-11 04:00:16,140 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=e892275aa04716c37d873e5baba493e7, ASSIGN in 358 msec 2023-02-11 04:00:16,142 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-11 04:00:16,142 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088016142"}]},"ts":"1676088016142"} 2023-02-11 04:00:16,144 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-02-11 04:00:16,149 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-02-11 04:00:16,151 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 534 msec 2023-02-11 04:00:16,227 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-02-11 04:00:16,229 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-02-11 04:00:16,229 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 04:00:16,262 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-02-11 04:00:16,278 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-11 04:00:16,284 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 29 msec 2023-02-11 04:00:16,295 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-02-11 04:00:16,306 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-11 04:00:16,311 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 15 msec 2023-02-11 04:00:16,320 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-11 04:00:16,323 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-11 04:00:16,323 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 2.500sec 2023-02-11 04:00:16,326 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-11 04:00:16,327 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-11 04:00:16,328 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-11 04:00:16,329 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,41713,1676088012566-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-11 04:00:16,330 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,41713,1676088012566-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-11 04:00:16,371 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-11 04:00:16,397 DEBUG [Listener at localhost/33753] zookeeper.ReadOnlyZKClient(139): Connect 0x0da5f7e5 to 127.0.0.1:63425 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 04:00:16,402 DEBUG [Listener at localhost/33753] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@492b920e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 04:00:16,416 DEBUG [hconnection-0x56d8d559-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 04:00:16,426 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55132, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 04:00:16,436 INFO [Listener at localhost/33753] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,41713,1676088012566 2023-02-11 04:00:16,464 INFO [Listener at localhost/33753] hbase.ResourceChecker(147): before: client.TestMobRestoreSnapshotFromClientSimple#testRestoreSnapshot Thread=358, OpenFileDescriptor=691, MaxFileDescriptor=60000, SystemLoadAverage=223, ProcessCount=170, AvailableMemoryMB=3209 2023-02-11 04:00:16,477 DEBUG [Listener at localhost/33753] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-11 04:00:16,481 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52380, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-11 04:00:16,492 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'testRestoreSnapshot-1676088016466', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2023-02-11 04:00:16,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:16,496 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676088016466 execute state=CREATE_TABLE_PRE_OPERATION 2023-02-11 04:00:16,499 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676088016466 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-11 04:00:16,501 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "testRestoreSnapshot-1676088016466" procId is: 9 2023-02-11 04:00:16,507 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:16,508 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:16,508 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:16,508 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:16,508 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:16,508 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:16,508 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539 empty. 2023-02-11 04:00:16,509 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e empty. 2023-02-11 04:00:16,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-11 04:00:16,511 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519 empty. 2023-02-11 04:00:16,512 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac empty. 2023-02-11 04:00:16,513 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4 empty. 2023-02-11 04:00:16,513 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:16,514 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808 empty. 2023-02-11 04:00:16,515 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:16,515 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:16,515 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:16,515 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:16,516 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:16,516 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676088016466 regions 2023-02-11 04:00:16,537 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/.tabledesc/.tableinfo.0000000001 2023-02-11 04:00:16,539 INFO [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-2] regionserver.HRegion(7671): creating {ENCODED => 51e4d6a6293430d21cdf75126c54f808, NAME => 'testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testRestoreSnapshot-1676088016466', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp 2023-02-11 04:00:16,539 INFO [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-1] regionserver.HRegion(7671): creating {ENCODED => 3452669615c81575c52ff96406a41e9e, NAME => 'testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testRestoreSnapshot-1676088016466', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp 2023-02-11 04:00:16,539 INFO [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-0] regionserver.HRegion(7671): creating {ENCODED => 747a6fa324aa972cea6bbd307f3b1539, NAME => 'testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testRestoreSnapshot-1676088016466', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp 2023-02-11 04:00:16,579 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:16,581 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-1] regionserver.HRegion(1603): Closing 3452669615c81575c52ff96406a41e9e, disabling compactions & flushes 2023-02-11 04:00:16,581 INFO [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:16,581 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:16,581 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. after waiting 0 ms 2023-02-11 04:00:16,581 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:16,581 INFO [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:16,581 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-1] regionserver.HRegion(1557): Region close journal for 3452669615c81575c52ff96406a41e9e: Waiting for close lock at 1676088016581Disabling compacts and flushes for region at 1676088016581Disabling writes for close at 1676088016581Writing region close event to WAL at 1676088016581Closed at 1676088016581 2023-02-11 04:00:16,583 INFO [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-1] regionserver.HRegion(7671): creating {ENCODED => e1d0ef015d92b032f67c5052a11865c4, NAME => 'testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testRestoreSnapshot-1676088016466', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp 2023-02-11 04:00:16,586 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:16,587 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-0] regionserver.HRegion(1603): Closing 747a6fa324aa972cea6bbd307f3b1539, disabling compactions & flushes 2023-02-11 04:00:16,587 INFO [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:16,587 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:16,587 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:16,587 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-2] regionserver.HRegion(1603): Closing 51e4d6a6293430d21cdf75126c54f808, disabling compactions & flushes 2023-02-11 04:00:16,588 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. after waiting 0 ms 2023-02-11 04:00:16,588 INFO [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:16,588 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:16,588 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:16,588 INFO [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:16,588 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. after waiting 0 ms 2023-02-11 04:00:16,588 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-0] regionserver.HRegion(1557): Region close journal for 747a6fa324aa972cea6bbd307f3b1539: Waiting for close lock at 1676088016587Disabling compacts and flushes for region at 1676088016587Disabling writes for close at 1676088016588 (+1 ms)Writing region close event to WAL at 1676088016588Closed at 1676088016588 2023-02-11 04:00:16,588 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:16,588 INFO [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:16,588 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-2] regionserver.HRegion(1557): Region close journal for 51e4d6a6293430d21cdf75126c54f808: Waiting for close lock at 1676088016587Disabling compacts and flushes for region at 1676088016587Disabling writes for close at 1676088016588 (+1 ms)Writing region close event to WAL at 1676088016588Closed at 1676088016588 2023-02-11 04:00:16,589 INFO [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-0] regionserver.HRegion(7671): creating {ENCODED => aaa51003a74d8671fa0840bbfcb549ac, NAME => 'testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testRestoreSnapshot-1676088016466', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp 2023-02-11 04:00:16,589 INFO [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-2] regionserver.HRegion(7671): creating {ENCODED => d586dffdbc612899aa5c72d99bdc6519, NAME => 'testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testRestoreSnapshot-1676088016466', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp 2023-02-11 04:00:16,614 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:16,615 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-1] regionserver.HRegion(1603): Closing e1d0ef015d92b032f67c5052a11865c4, disabling compactions & flushes 2023-02-11 04:00:16,615 INFO [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:16,615 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:16,615 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. after waiting 0 ms 2023-02-11 04:00:16,615 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:16,615 INFO [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:16,615 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-1] regionserver.HRegion(1557): Region close journal for e1d0ef015d92b032f67c5052a11865c4: Waiting for close lock at 1676088016615Disabling compacts and flushes for region at 1676088016615Disabling writes for close at 1676088016615Writing region close event to WAL at 1676088016615Closed at 1676088016615 2023-02-11 04:00:16,615 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:16,617 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-2] regionserver.HRegion(1603): Closing d586dffdbc612899aa5c72d99bdc6519, disabling compactions & flushes 2023-02-11 04:00:16,617 INFO [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:16,617 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:16,617 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. after waiting 0 ms 2023-02-11 04:00:16,617 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:16,617 INFO [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:16,617 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-2] regionserver.HRegion(1557): Region close journal for d586dffdbc612899aa5c72d99bdc6519: Waiting for close lock at 1676088016617Disabling compacts and flushes for region at 1676088016617Disabling writes for close at 1676088016617Writing region close event to WAL at 1676088016617Closed at 1676088016617 2023-02-11 04:00:16,617 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:16,618 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-0] regionserver.HRegion(1603): Closing aaa51003a74d8671fa0840bbfcb549ac, disabling compactions & flushes 2023-02-11 04:00:16,618 INFO [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:16,618 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:16,618 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. after waiting 0 ms 2023-02-11 04:00:16,618 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:16,618 INFO [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:16,618 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676088016466-pool-0] regionserver.HRegion(1557): Region close journal for aaa51003a74d8671fa0840bbfcb549ac: Waiting for close lock at 1676088016618Disabling compacts and flushes for region at 1676088016618Disabling writes for close at 1676088016618Writing region close event to WAL at 1676088016618Closed at 1676088016618 2023-02-11 04:00:16,622 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676088016466 execute state=CREATE_TABLE_ADD_TO_META 2023-02-11 04:00:16,624 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088016624"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088016624"}]},"ts":"1676088016624"} 2023-02-11 04:00:16,624 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088016624"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088016624"}]},"ts":"1676088016624"} 2023-02-11 04:00:16,624 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088016624"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088016624"}]},"ts":"1676088016624"} 2023-02-11 04:00:16,624 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088016624"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088016624"}]},"ts":"1676088016624"} 2023-02-11 04:00:16,625 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088016624"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088016624"}]},"ts":"1676088016624"} 2023-02-11 04:00:16,625 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088016624"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088016624"}]},"ts":"1676088016624"} 2023-02-11 04:00:16,669 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-11 04:00:16,671 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676088016466 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-11 04:00:16,671 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088016671"}]},"ts":"1676088016671"} 2023-02-11 04:00:16,673 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676088016466, state=ENABLING in hbase:meta 2023-02-11 04:00:16,678 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 04:00:16,683 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 04:00:16,683 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 04:00:16,683 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 04:00:16,683 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 04:00:16,684 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, ASSIGN}, {pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, ASSIGN}, {pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, ASSIGN}, {pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, ASSIGN}, {pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, ASSIGN}, {pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, ASSIGN}] 2023-02-11 04:00:16,687 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, ASSIGN 2023-02-11 04:00:16,687 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, ASSIGN 2023-02-11 04:00:16,688 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, ASSIGN 2023-02-11 04:00:16,688 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, ASSIGN 2023-02-11 04:00:16,689 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42385,1676088013771; forceNewPlan=false, retain=false 2023-02-11 04:00:16,689 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40829,1676088013739; forceNewPlan=false, retain=false 2023-02-11 04:00:16,689 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40829,1676088013739; forceNewPlan=false, retain=false 2023-02-11 04:00:16,690 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46851,1676088013694; forceNewPlan=false, retain=false 2023-02-11 04:00:16,690 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, ASSIGN 2023-02-11 04:00:16,691 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, ASSIGN 2023-02-11 04:00:16,691 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42385,1676088013771; forceNewPlan=false, retain=false 2023-02-11 04:00:16,692 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46851,1676088013694; forceNewPlan=false, retain=false 2023-02-11 04:00:16,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-11 04:00:16,840 INFO [jenkins-hbase4:41713] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-11 04:00:16,842 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=aaa51003a74d8671fa0840bbfcb549ac, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:16,842 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=3452669615c81575c52ff96406a41e9e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:16,842 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=d586dffdbc612899aa5c72d99bdc6519, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:16,842 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088016842"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088016842"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088016842"}]},"ts":"1676088016842"} 2023-02-11 04:00:16,842 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=747a6fa324aa972cea6bbd307f3b1539, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:16,843 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088016842"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088016842"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088016842"}]},"ts":"1676088016842"} 2023-02-11 04:00:16,842 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=51e4d6a6293430d21cdf75126c54f808, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:16,843 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088016842"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088016842"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088016842"}]},"ts":"1676088016842"} 2023-02-11 04:00:16,842 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088016842"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088016842"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088016842"}]},"ts":"1676088016842"} 2023-02-11 04:00:16,843 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088016842"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088016842"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088016842"}]},"ts":"1676088016842"} 2023-02-11 04:00:16,845 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=14, state=RUNNABLE; OpenRegionProcedure aaa51003a74d8671fa0840bbfcb549ac, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:16,847 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=15, state=RUNNABLE; OpenRegionProcedure d586dffdbc612899aa5c72d99bdc6519, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:16,849 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=10, state=RUNNABLE; OpenRegionProcedure 747a6fa324aa972cea6bbd307f3b1539, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:16,851 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=11, state=RUNNABLE; OpenRegionProcedure 3452669615c81575c52ff96406a41e9e, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:16,851 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=e1d0ef015d92b032f67c5052a11865c4, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:16,852 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088016851"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088016851"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088016851"}]},"ts":"1676088016851"} 2023-02-11 04:00:16,853 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=12, state=RUNNABLE; OpenRegionProcedure 51e4d6a6293430d21cdf75126c54f808, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:16,855 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=13, state=RUNNABLE; OpenRegionProcedure e1d0ef015d92b032f67c5052a11865c4, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:16,999 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:16,999 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-11 04:00:17,000 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:17,000 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-11 04:00:17,003 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34706, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-11 04:00:17,003 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49616, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-11 04:00:17,010 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:17,011 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:17,011 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:17,011 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 747a6fa324aa972cea6bbd307f3b1539, NAME => 'testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 04:00:17,011 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => aaa51003a74d8671fa0840bbfcb549ac, NAME => 'testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 04:00:17,011 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e1d0ef015d92b032f67c5052a11865c4, NAME => 'testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 04:00:17,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:17,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:17,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:17,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:17,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:17,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:17,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:17,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:17,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:17,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:17,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:17,013 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:17,016 INFO [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:17,016 INFO [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:17,016 INFO [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:17,018 DEBUG [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf 2023-02-11 04:00:17,018 DEBUG [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf 2023-02-11 04:00:17,018 DEBUG [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf 2023-02-11 04:00:17,018 DEBUG [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf 2023-02-11 04:00:17,020 DEBUG [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf 2023-02-11 04:00:17,020 DEBUG [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf 2023-02-11 04:00:17,021 INFO [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e1d0ef015d92b032f67c5052a11865c4 columnFamilyName cf 2023-02-11 04:00:17,021 INFO [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region aaa51003a74d8671fa0840bbfcb549ac columnFamilyName cf 2023-02-11 04:00:17,021 INFO [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 747a6fa324aa972cea6bbd307f3b1539 columnFamilyName cf 2023-02-11 04:00:17,024 INFO [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] regionserver.HStore(310): Store=e1d0ef015d92b032f67c5052a11865c4/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:17,028 INFO [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] regionserver.HStore(310): Store=747a6fa324aa972cea6bbd307f3b1539/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:17,028 INFO [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] regionserver.HStore(310): Store=aaa51003a74d8671fa0840bbfcb549ac/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:17,032 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:17,032 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:17,032 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:17,033 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:17,033 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:17,033 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:17,038 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:17,038 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:17,044 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:17,049 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 04:00:17,050 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 04:00:17,050 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 04:00:17,051 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 747a6fa324aa972cea6bbd307f3b1539; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63840528, jitterRate=-0.04870200157165527}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:17,051 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e1d0ef015d92b032f67c5052a11865c4; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67676181, jitterRate=0.00845368206501007}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:17,051 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 747a6fa324aa972cea6bbd307f3b1539: Running coprocessor pre-open hook at 1676088017013Writing region info on filesystem at 1676088017013Initializing all the Stores at 1676088017014 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088017014Cleaning up temporary data from old regions at 1676088017033 (+19 ms)Cleaning up detritus from prior splits at 1676088017034 (+1 ms)Running coprocessor post-open hooks at 1676088017051 (+17 ms)Region opened successfully at 1676088017051 2023-02-11 04:00:17,051 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e1d0ef015d92b032f67c5052a11865c4: Running coprocessor pre-open hook at 1676088017013Writing region info on filesystem at 1676088017013Initializing all the Stores at 1676088017014 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088017015 (+1 ms)Cleaning up temporary data from old regions at 1676088017034 (+19 ms)Cleaning up detritus from prior splits at 1676088017035 (+1 ms)Running coprocessor post-open hooks at 1676088017051 (+16 ms)Region opened successfully at 1676088017051 2023-02-11 04:00:17,051 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened aaa51003a74d8671fa0840bbfcb549ac; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66002602, jitterRate=-0.016484588384628296}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:17,052 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for aaa51003a74d8671fa0840bbfcb549ac: Running coprocessor pre-open hook at 1676088017013Writing region info on filesystem at 1676088017013Initializing all the Stores at 1676088017014 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088017014Cleaning up temporary data from old regions at 1676088017034 (+20 ms)Cleaning up detritus from prior splits at 1676088017034Running coprocessor post-open hooks at 1676088017052 (+18 ms)Region opened successfully at 1676088017052 2023-02-11 04:00:17,053 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4., pid=21, masterSystemTime=1676088017004 2023-02-11 04:00:17,053 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539., pid=18, masterSystemTime=1676088017000 2023-02-11 04:00:17,055 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac., pid=16, masterSystemTime=1676088016999 2023-02-11 04:00:17,059 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:17,059 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:17,059 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:17,059 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 3452669615c81575c52ff96406a41e9e, NAME => 'testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 04:00:17,060 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=e1d0ef015d92b032f67c5052a11865c4, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:17,060 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:17,060 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:17,060 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:17,060 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:17,061 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:17,061 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:17,060 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088017060"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088017060"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088017060"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088017060"}]},"ts":"1676088017060"} 2023-02-11 04:00:17,061 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:17,061 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d586dffdbc612899aa5c72d99bdc6519, NAME => 'testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 04:00:17,061 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=747a6fa324aa972cea6bbd307f3b1539, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:17,062 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088017061"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088017061"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088017061"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088017061"}]},"ts":"1676088017061"} 2023-02-11 04:00:17,062 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:17,062 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:17,062 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:17,062 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:17,063 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:17,064 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:17,064 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:17,064 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 51e4d6a6293430d21cdf75126c54f808, NAME => 'testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 04:00:17,064 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:17,064 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:17,064 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:17,064 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:17,065 INFO [StoreOpener-3452669615c81575c52ff96406a41e9e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:17,065 INFO [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:17,066 INFO [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:17,067 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=aaa51003a74d8671fa0840bbfcb549ac, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:17,068 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088017067"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088017067"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088017067"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088017067"}]},"ts":"1676088017067"} 2023-02-11 04:00:17,069 DEBUG [StoreOpener-3452669615c81575c52ff96406a41e9e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf 2023-02-11 04:00:17,069 DEBUG [StoreOpener-3452669615c81575c52ff96406a41e9e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf 2023-02-11 04:00:17,069 DEBUG [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf 2023-02-11 04:00:17,070 DEBUG [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf 2023-02-11 04:00:17,071 INFO [StoreOpener-3452669615c81575c52ff96406a41e9e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3452669615c81575c52ff96406a41e9e columnFamilyName cf 2023-02-11 04:00:17,071 INFO [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d586dffdbc612899aa5c72d99bdc6519 columnFamilyName cf 2023-02-11 04:00:17,071 DEBUG [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf 2023-02-11 04:00:17,072 DEBUG [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf 2023-02-11 04:00:17,072 INFO [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 51e4d6a6293430d21cdf75126c54f808 columnFamilyName cf 2023-02-11 04:00:17,073 INFO [StoreOpener-3452669615c81575c52ff96406a41e9e-1] regionserver.HStore(310): Store=3452669615c81575c52ff96406a41e9e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:17,074 INFO [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] regionserver.HStore(310): Store=d586dffdbc612899aa5c72d99bdc6519/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:17,077 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:17,077 INFO [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] regionserver.HStore(310): Store=51e4d6a6293430d21cdf75126c54f808/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:17,078 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:17,078 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=13 2023-02-11 04:00:17,079 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=13, state=SUCCESS; OpenRegionProcedure e1d0ef015d92b032f67c5052a11865c4, server=jenkins-hbase4.apache.org,40829,1676088013739 in 213 msec 2023-02-11 04:00:17,079 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:17,079 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:17,080 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:17,081 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=10 2023-02-11 04:00:17,081 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=10, state=SUCCESS; OpenRegionProcedure 747a6fa324aa972cea6bbd307f3b1539, server=jenkins-hbase4.apache.org,42385,1676088013771 in 223 msec 2023-02-11 04:00:17,081 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:17,083 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, ASSIGN in 395 msec 2023-02-11 04:00:17,086 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=14 2023-02-11 04:00:17,086 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=14, state=SUCCESS; OpenRegionProcedure aaa51003a74d8671fa0840bbfcb549ac, server=jenkins-hbase4.apache.org,46851,1676088013694 in 231 msec 2023-02-11 04:00:17,087 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, ASSIGN in 397 msec 2023-02-11 04:00:17,087 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:17,088 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:17,089 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:17,089 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, ASSIGN in 402 msec 2023-02-11 04:00:17,092 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 04:00:17,092 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 04:00:17,093 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 04:00:17,093 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 3452669615c81575c52ff96406a41e9e; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75492306, jitterRate=0.12492302060127258}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:17,093 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened d586dffdbc612899aa5c72d99bdc6519; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72270047, jitterRate=0.07690761983394623}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:17,093 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 3452669615c81575c52ff96406a41e9e: Running coprocessor pre-open hook at 1676088017061Writing region info on filesystem at 1676088017061Initializing all the Stores at 1676088017064 (+3 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088017064Cleaning up temporary data from old regions at 1676088017080 (+16 ms)Cleaning up detritus from prior splits at 1676088017081 (+1 ms)Running coprocessor post-open hooks at 1676088017093 (+12 ms)Region opened successfully at 1676088017093 2023-02-11 04:00:17,093 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for d586dffdbc612899aa5c72d99bdc6519: Running coprocessor pre-open hook at 1676088017063Writing region info on filesystem at 1676088017063Initializing all the Stores at 1676088017065 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088017065Cleaning up temporary data from old regions at 1676088017080 (+15 ms)Cleaning up detritus from prior splits at 1676088017081 (+1 ms)Running coprocessor post-open hooks at 1676088017093 (+12 ms)Region opened successfully at 1676088017093 2023-02-11 04:00:17,093 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 51e4d6a6293430d21cdf75126c54f808; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62906981, jitterRate=-0.06261293590068817}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:17,094 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 51e4d6a6293430d21cdf75126c54f808: Running coprocessor pre-open hook at 1676088017065Writing region info on filesystem at 1676088017065Initializing all the Stores at 1676088017066 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088017066Cleaning up temporary data from old regions at 1676088017082 (+16 ms)Cleaning up detritus from prior splits at 1676088017085 (+3 ms)Running coprocessor post-open hooks at 1676088017094 (+9 ms)Region opened successfully at 1676088017094 2023-02-11 04:00:17,095 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519., pid=17, masterSystemTime=1676088017000 2023-02-11 04:00:17,096 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e., pid=19, masterSystemTime=1676088017004 2023-02-11 04:00:17,096 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808., pid=20, masterSystemTime=1676088016999 2023-02-11 04:00:17,098 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:17,099 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:17,100 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=d586dffdbc612899aa5c72d99bdc6519, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:17,101 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088017100"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088017100"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088017100"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088017100"}]},"ts":"1676088017100"} 2023-02-11 04:00:17,102 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=3452669615c81575c52ff96406a41e9e, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:17,102 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088017102"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088017102"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088017102"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088017102"}]},"ts":"1676088017102"} 2023-02-11 04:00:17,103 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:17,103 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:17,104 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:17,104 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:17,105 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=51e4d6a6293430d21cdf75126c54f808, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:17,105 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088017104"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088017104"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088017104"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088017104"}]},"ts":"1676088017104"} 2023-02-11 04:00:17,108 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=15 2023-02-11 04:00:17,108 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=15, state=SUCCESS; OpenRegionProcedure d586dffdbc612899aa5c72d99bdc6519, server=jenkins-hbase4.apache.org,42385,1676088013771 in 258 msec 2023-02-11 04:00:17,111 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=11 2023-02-11 04:00:17,111 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, ASSIGN in 424 msec 2023-02-11 04:00:17,112 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=11, state=SUCCESS; OpenRegionProcedure 3452669615c81575c52ff96406a41e9e, server=jenkins-hbase4.apache.org,40829,1676088013739 in 256 msec 2023-02-11 04:00:17,113 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=12 2023-02-11 04:00:17,113 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=12, state=SUCCESS; OpenRegionProcedure 51e4d6a6293430d21cdf75126c54f808, server=jenkins-hbase4.apache.org,46851,1676088013694 in 255 msec 2023-02-11 04:00:17,114 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, ASSIGN in 427 msec 2023-02-11 04:00:17,116 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=12, resume processing ppid=9 2023-02-11 04:00:17,116 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=12, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, ASSIGN in 429 msec 2023-02-11 04:00:17,117 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676088016466 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-11 04:00:17,118 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088017118"}]},"ts":"1676088017118"} 2023-02-11 04:00:17,120 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676088016466, state=ENABLED in hbase:meta 2023-02-11 04:00:17,122 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676088016466 execute state=CREATE_TABLE_POST_OPERATION 2023-02-11 04:00:17,125 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testRestoreSnapshot-1676088016466 in 631 msec 2023-02-11 04:00:17,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-11 04:00:17,265 INFO [Listener at localhost/33753] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testRestoreSnapshot-1676088016466, procId: 9 completed 2023-02-11 04:00:17,272 DEBUG [Listener at localhost/33753] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676088016466 2023-02-11 04:00:17,273 DEBUG [Listener at localhost/33753] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:17,276 INFO [Listener at localhost/33753] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-11 04:00:17,298 DEBUG [hconnection-0x56d8d559-shared-pool-4] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 04:00:17,301 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49628, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 04:00:17,305 DEBUG [hconnection-0x56d8d559-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 04:00:17,308 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34714, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 04:00:17,320 INFO [Listener at localhost/33753] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676088016466 2023-02-11 04:00:17,324 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676088016466 2023-02-11 04:00:17,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] procedure2.ProcedureExecutor(1029): Stored pid=22, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:17,336 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088017336"}]},"ts":"1676088017336"} 2023-02-11 04:00:17,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-11 04:00:17,339 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676088016466, state=DISABLING in hbase:meta 2023-02-11 04:00:17,341 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676088016466 to state=DISABLING 2023-02-11 04:00:17,343 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, UNASSIGN}, {pid=24, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, UNASSIGN}, {pid=25, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, UNASSIGN}, {pid=26, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, UNASSIGN}, {pid=27, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, UNASSIGN}, {pid=28, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, UNASSIGN}] 2023-02-11 04:00:17,346 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, UNASSIGN 2023-02-11 04:00:17,346 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, UNASSIGN 2023-02-11 04:00:17,346 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=23, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, UNASSIGN 2023-02-11 04:00:17,346 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, UNASSIGN 2023-02-11 04:00:17,347 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, UNASSIGN 2023-02-11 04:00:17,348 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=51e4d6a6293430d21cdf75126c54f808, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:17,348 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088017348"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088017348"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088017348"}]},"ts":"1676088017348"} 2023-02-11 04:00:17,349 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=3452669615c81575c52ff96406a41e9e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:17,349 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=e1d0ef015d92b032f67c5052a11865c4, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:17,349 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088017349"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088017349"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088017349"}]},"ts":"1676088017349"} 2023-02-11 04:00:17,349 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088017349"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088017349"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088017349"}]},"ts":"1676088017349"} 2023-02-11 04:00:17,349 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=d586dffdbc612899aa5c72d99bdc6519, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:17,349 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=747a6fa324aa972cea6bbd307f3b1539, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:17,349 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088017349"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088017349"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088017349"}]},"ts":"1676088017349"} 2023-02-11 04:00:17,350 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088017349"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088017349"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088017349"}]},"ts":"1676088017349"} 2023-02-11 04:00:17,351 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=25, state=RUNNABLE; CloseRegionProcedure 51e4d6a6293430d21cdf75126c54f808, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:17,353 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=24, state=RUNNABLE; CloseRegionProcedure 3452669615c81575c52ff96406a41e9e, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:17,354 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=31, ppid=26, state=RUNNABLE; CloseRegionProcedure e1d0ef015d92b032f67c5052a11865c4, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:17,356 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=28, state=RUNNABLE; CloseRegionProcedure d586dffdbc612899aa5c72d99bdc6519, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:17,358 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=23, state=RUNNABLE; CloseRegionProcedure 747a6fa324aa972cea6bbd307f3b1539, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:17,364 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=27, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, UNASSIGN 2023-02-11 04:00:17,365 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=aaa51003a74d8671fa0840bbfcb549ac, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:17,365 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088017365"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088017365"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088017365"}]},"ts":"1676088017365"} 2023-02-11 04:00:17,367 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=34, ppid=27, state=RUNNABLE; CloseRegionProcedure aaa51003a74d8671fa0840bbfcb549ac, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:17,517 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:17,517 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:17,517 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing d586dffdbc612899aa5c72d99bdc6519, disabling compactions & flushes 2023-02-11 04:00:17,517 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:17,518 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:17,518 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 51e4d6a6293430d21cdf75126c54f808, disabling compactions & flushes 2023-02-11 04:00:17,518 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 3452669615c81575c52ff96406a41e9e, disabling compactions & flushes 2023-02-11 04:00:17,518 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:17,518 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:17,518 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:17,518 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. after waiting 0 ms 2023-02-11 04:00:17,518 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:17,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:17,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:17,518 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. after waiting 0 ms 2023-02-11 04:00:17,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. after waiting 0 ms 2023-02-11 04:00:17,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:17,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:17,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 04:00:17,528 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 04:00:17,528 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:17,528 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 3452669615c81575c52ff96406a41e9e: Waiting for close lock at 1676088017517Running coprocessor pre-close hooks at 1676088017517Disabling compacts and flushes for region at 1676088017517Disabling writes for close at 1676088017519 (+2 ms)Writing region close event to WAL at 1676088017519Running coprocessor post-close hooks at 1676088017528 (+9 ms)Closed at 1676088017528 2023-02-11 04:00:17,528 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 04:00:17,529 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:17,529 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for d586dffdbc612899aa5c72d99bdc6519: Waiting for close lock at 1676088017517Running coprocessor pre-close hooks at 1676088017517Disabling compacts and flushes for region at 1676088017517Disabling writes for close at 1676088017519 (+2 ms)Writing region close event to WAL at 1676088017519Running coprocessor post-close hooks at 1676088017529 (+10 ms)Closed at 1676088017529 2023-02-11 04:00:17,529 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:17,530 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 51e4d6a6293430d21cdf75126c54f808: Waiting for close lock at 1676088017518Running coprocessor pre-close hooks at 1676088017518Disabling compacts and flushes for region at 1676088017518Disabling writes for close at 1676088017519 (+1 ms)Writing region close event to WAL at 1676088017519Running coprocessor post-close hooks at 1676088017529 (+10 ms)Closed at 1676088017529 2023-02-11 04:00:17,531 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:17,531 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:17,531 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e1d0ef015d92b032f67c5052a11865c4, disabling compactions & flushes 2023-02-11 04:00:17,531 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:17,531 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:17,531 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. after waiting 0 ms 2023-02-11 04:00:17,531 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:17,532 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=3452669615c81575c52ff96406a41e9e, regionState=CLOSED 2023-02-11 04:00:17,533 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088017532"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088017532"}]},"ts":"1676088017532"} 2023-02-11 04:00:17,533 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:17,533 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:17,533 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 747a6fa324aa972cea6bbd307f3b1539, disabling compactions & flushes 2023-02-11 04:00:17,533 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:17,533 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:17,533 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. after waiting 0 ms 2023-02-11 04:00:17,533 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:17,536 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=d586dffdbc612899aa5c72d99bdc6519, regionState=CLOSED 2023-02-11 04:00:17,536 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088017536"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088017536"}]},"ts":"1676088017536"} 2023-02-11 04:00:17,537 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:17,537 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:17,537 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=51e4d6a6293430d21cdf75126c54f808, regionState=CLOSED 2023-02-11 04:00:17,538 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088017537"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088017537"}]},"ts":"1676088017537"} 2023-02-11 04:00:17,540 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 04:00:17,545 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing aaa51003a74d8671fa0840bbfcb549ac, disabling compactions & flushes 2023-02-11 04:00:17,545 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 04:00:17,545 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:17,545 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:17,545 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e1d0ef015d92b032f67c5052a11865c4: Waiting for close lock at 1676088017531Running coprocessor pre-close hooks at 1676088017531Disabling compacts and flushes for region at 1676088017531Disabling writes for close at 1676088017531Writing region close event to WAL at 1676088017532 (+1 ms)Running coprocessor post-close hooks at 1676088017545 (+13 ms)Closed at 1676088017545 2023-02-11 04:00:17,545 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:17,545 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. after waiting 0 ms 2023-02-11 04:00:17,545 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:17,548 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:17,548 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:17,548 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 747a6fa324aa972cea6bbd307f3b1539: Waiting for close lock at 1676088017533Running coprocessor pre-close hooks at 1676088017533Disabling compacts and flushes for region at 1676088017533Disabling writes for close at 1676088017533Writing region close event to WAL at 1676088017534 (+1 ms)Running coprocessor post-close hooks at 1676088017548 (+14 ms)Closed at 1676088017548 2023-02-11 04:00:17,549 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=e1d0ef015d92b032f67c5052a11865c4, regionState=CLOSED 2023-02-11 04:00:17,549 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088017549"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088017549"}]},"ts":"1676088017549"} 2023-02-11 04:00:17,551 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=24 2023-02-11 04:00:17,551 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=24, state=SUCCESS; CloseRegionProcedure 3452669615c81575c52ff96406a41e9e, server=jenkins-hbase4.apache.org,40829,1676088013739 in 192 msec 2023-02-11 04:00:17,552 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=28 2023-02-11 04:00:17,552 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:17,552 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=28, state=SUCCESS; CloseRegionProcedure d586dffdbc612899aa5c72d99bdc6519, server=jenkins-hbase4.apache.org,42385,1676088013771 in 191 msec 2023-02-11 04:00:17,554 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 04:00:17,554 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=747a6fa324aa972cea6bbd307f3b1539, regionState=CLOSED 2023-02-11 04:00:17,554 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088017554"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088017554"}]},"ts":"1676088017554"} 2023-02-11 04:00:17,555 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:17,555 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for aaa51003a74d8671fa0840bbfcb549ac: Waiting for close lock at 1676088017537Running coprocessor pre-close hooks at 1676088017537Disabling compacts and flushes for region at 1676088017537Disabling writes for close at 1676088017545 (+8 ms)Writing region close event to WAL at 1676088017546 (+1 ms)Running coprocessor post-close hooks at 1676088017555 (+9 ms)Closed at 1676088017555 2023-02-11 04:00:17,556 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, UNASSIGN in 208 msec 2023-02-11 04:00:17,556 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=25 2023-02-11 04:00:17,556 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, UNASSIGN in 209 msec 2023-02-11 04:00:17,556 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=25, state=SUCCESS; CloseRegionProcedure 51e4d6a6293430d21cdf75126c54f808, server=jenkins-hbase4.apache.org,46851,1676088013694 in 198 msec 2023-02-11 04:00:17,560 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=31, resume processing ppid=26 2023-02-11 04:00:17,560 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=26, state=SUCCESS; CloseRegionProcedure e1d0ef015d92b032f67c5052a11865c4, server=jenkins-hbase4.apache.org,40829,1676088013739 in 198 msec 2023-02-11 04:00:17,560 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:17,561 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=aaa51003a74d8671fa0840bbfcb549ac, regionState=CLOSED 2023-02-11 04:00:17,561 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, UNASSIGN in 213 msec 2023-02-11 04:00:17,561 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088017561"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088017561"}]},"ts":"1676088017561"} 2023-02-11 04:00:17,563 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, UNASSIGN in 218 msec 2023-02-11 04:00:17,564 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=23 2023-02-11 04:00:17,564 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=23, state=SUCCESS; CloseRegionProcedure 747a6fa324aa972cea6bbd307f3b1539, server=jenkins-hbase4.apache.org,42385,1676088013771 in 202 msec 2023-02-11 04:00:17,566 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, UNASSIGN in 221 msec 2023-02-11 04:00:17,566 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=34, resume processing ppid=27 2023-02-11 04:00:17,566 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=34, ppid=27, state=SUCCESS; CloseRegionProcedure aaa51003a74d8671fa0840bbfcb549ac, server=jenkins-hbase4.apache.org,46851,1676088013694 in 196 msec 2023-02-11 04:00:17,569 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=22 2023-02-11 04:00:17,569 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, UNASSIGN in 223 msec 2023-02-11 04:00:17,570 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088017570"}]},"ts":"1676088017570"} 2023-02-11 04:00:17,572 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676088016466, state=DISABLED in hbase:meta 2023-02-11 04:00:17,574 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676088016466 to state=DISABLED 2023-02-11 04:00:17,577 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676088016466 in 249 msec 2023-02-11 04:00:17,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-11 04:00:17,589 INFO [Listener at localhost/33753] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676088016466, procId: 22 completed 2023-02-11 04:00:17,599 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=emptySnaptb-1676088016466 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 } 2023-02-11 04:00:17,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676088017599 (current time:1676088017599). 2023-02-11 04:00:17,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-11 04:00:17,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] zookeeper.ReadOnlyZKClient(139): Connect 0x608a2ef1 to 127.0.0.1:63425 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 04:00:17,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3e05431, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 04:00:17,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 04:00:17,612 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55142, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 04:00:17,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x608a2ef1 to 127.0.0.1:63425 2023-02-11 04:00:17,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:00:17,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-11 04:00:17,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=emptySnaptb-1676088016466 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 } 2023-02-11 04:00:17,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676088016466, type=EXCLUSIVE 2023-02-11 04:00:17,637 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=35, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676088016466, type=EXCLUSIVE 2023-02-11 04:00:17,639 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=35, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676088016466, type=EXCLUSIVE; timeout=600000, timestamp=1676088617638 2023-02-11 04:00:17,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(664): Started snapshot: { ss=emptySnaptb-1676088016466 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 } 2023-02-11 04:00:17,639 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot emptySnaptb-1676088016466 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676088016466 2023-02-11 04:00:17,644 DEBUG [Listener at localhost/33753] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=emptySnaptb-1676088016466 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-11 04:00:17,644 DEBUG [Listener at localhost/33753] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-11 04:00:17,659 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676088016466 table=testRestoreSnapshot-1676088016466 type=DISABLED ttl=0 } 2023-02-11 04:00:17,662 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:17,662 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:17,662 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:17,663 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:17,663 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:17,663 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:17,663 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:17,667 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:17,667 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:17,667 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:17,667 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:17,667 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:17,667 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:17,667 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:17,675 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 04:00:17,675 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 04:00:17,675 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 04:00:17,675 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 04:00:17,675 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 04:00:17,676 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 04:00:17,679 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 04:00:17,742 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=emptySnaptb-1676088016466 table=testRestoreSnapshot-1676088016466 type=DISABLED ttl=0 } as finished. 2023-02-11 04:00:17,742 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for emptySnaptb-1676088016466 2023-02-11 04:00:17,744 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/.tmp/emptySnaptb-1676088016466 2023-02-11 04:00:17,785 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/.tmp/emptySnaptb-1676088016466 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/emptySnaptb-1676088016466 2023-02-11 04:00:17,817 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot emptySnaptb-1676088016466 of table testRestoreSnapshot-1676088016466 completed 2023-02-11 04:00:17,818 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/.tmp/emptySnaptb-1676088016466 2023-02-11 04:00:17,818 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/.tmp/emptySnaptb-1676088016466 2023-02-11 04:00:17,818 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot emptySnaptb-1676088016466 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676088016466 at 1676088017639Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676088016466 table=testRestoreSnapshot-1676088016466 type=DISABLED ttl=0 } at 1676088017659 (+20 ms)Consolidate snapshot: emptySnaptb-1676088016466 at 1676088017742 (+83 ms)Loading Region manifests for emptySnaptb-1676088016466 at 1676088017742Writing data manifest for emptySnaptb-1676088016466 at 1676088017758 (+16 ms)Verifying snapshot: emptySnaptb-1676088016466 at 1676088017773 (+15 ms)Snapshot emptySnaptb-1676088016466 of table testRestoreSnapshot-1676088016466 completed at 1676088017817 (+44 ms) 2023-02-11 04:00:17,819 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=35, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676088016466, type=EXCLUSIVE 2023-02-11 04:00:17,822 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676088016466, type=EXCLUSIVE in 190 msec 2023-02-11 04:00:17,894 DEBUG [Listener at localhost/33753] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 04:00:17,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676088016466 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 } is done 2023-02-11 04:00:17,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(401): Snapshot '{ ss=emptySnaptb-1676088016466 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-11 04:00:17,900 INFO [Listener at localhost/33753] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676088016466 2023-02-11 04:00:17,904 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676088016466 2023-02-11 04:00:17,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] procedure2.ProcedureExecutor(1029): Stored pid=36, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:17,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-11 04:00:17,914 INFO [PEWorker-4] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676088016466 2023-02-11 04:00:17,914 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088017914"}]},"ts":"1676088017914"} 2023-02-11 04:00:17,916 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676088016466, state=ENABLING in hbase:meta 2023-02-11 04:00:17,919 DEBUG [PEWorker-4] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-11 04:00:17,922 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, ASSIGN}, {pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, ASSIGN}, {pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, ASSIGN}, {pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, ASSIGN}, {pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, ASSIGN}, {pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, ASSIGN}] 2023-02-11 04:00:17,924 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, ASSIGN 2023-02-11 04:00:17,924 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, ASSIGN 2023-02-11 04:00:17,924 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, ASSIGN 2023-02-11 04:00:17,924 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, ASSIGN 2023-02-11 04:00:17,925 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, ASSIGN 2023-02-11 04:00:17,925 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:17,926 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:17,926 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:17,926 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,40829,1676088013739; forceNewPlan=false, retain=true 2023-02-11 04:00:17,926 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,46851,1676088013694; forceNewPlan=false, retain=true 2023-02-11 04:00:17,926 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:17,926 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,42385,1676088013771; forceNewPlan=false, retain=true 2023-02-11 04:00:17,926 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,46851,1676088013694; forceNewPlan=false, retain=true 2023-02-11 04:00:17,926 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:17,927 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,40829,1676088013739; forceNewPlan=false, retain=true 2023-02-11 04:00:17,927 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, ASSIGN 2023-02-11 04:00:17,928 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:17,928 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,42385,1676088013771; forceNewPlan=false, retain=true 2023-02-11 04:00:18,076 INFO [jenkins-hbase4:41713] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-11 04:00:18,076 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=d586dffdbc612899aa5c72d99bdc6519, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:18,076 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=41 updating hbase:meta row=aaa51003a74d8671fa0840bbfcb549ac, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:18,076 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=747a6fa324aa972cea6bbd307f3b1539, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:18,076 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=39 updating hbase:meta row=51e4d6a6293430d21cdf75126c54f808, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:18,076 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=38 updating hbase:meta row=3452669615c81575c52ff96406a41e9e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:18,077 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088018076"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088018076"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088018076"}]},"ts":"1676088018076"} 2023-02-11 04:00:18,077 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088018076"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088018076"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088018076"}]},"ts":"1676088018076"} 2023-02-11 04:00:18,077 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088018076"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088018076"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088018076"}]},"ts":"1676088018076"} 2023-02-11 04:00:18,077 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088018076"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088018076"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088018076"}]},"ts":"1676088018076"} 2023-02-11 04:00:18,077 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088018076"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088018076"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088018076"}]},"ts":"1676088018076"} 2023-02-11 04:00:18,079 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=43, ppid=42, state=RUNNABLE; OpenRegionProcedure d586dffdbc612899aa5c72d99bdc6519, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:18,081 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=38, state=RUNNABLE; OpenRegionProcedure 3452669615c81575c52ff96406a41e9e, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:18,082 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=41, state=RUNNABLE; OpenRegionProcedure aaa51003a74d8671fa0840bbfcb549ac, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:18,083 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=46, ppid=37, state=RUNNABLE; OpenRegionProcedure 747a6fa324aa972cea6bbd307f3b1539, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:18,084 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=47, ppid=39, state=RUNNABLE; OpenRegionProcedure 51e4d6a6293430d21cdf75126c54f808, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:18,084 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=e1d0ef015d92b032f67c5052a11865c4, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:18,085 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088018084"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088018084"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088018084"}]},"ts":"1676088018084"} 2023-02-11 04:00:18,087 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=40, state=RUNNABLE; OpenRegionProcedure e1d0ef015d92b032f67c5052a11865c4, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:18,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-11 04:00:18,237 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:18,237 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:18,237 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 747a6fa324aa972cea6bbd307f3b1539, NAME => 'testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 04:00:18,237 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e1d0ef015d92b032f67c5052a11865c4, NAME => 'testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 04:00:18,238 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:18,238 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:18,238 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:18,238 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:18,238 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:18,238 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:18,238 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:18,238 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:18,240 INFO [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:18,240 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:18,240 INFO [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:18,240 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 51e4d6a6293430d21cdf75126c54f808, NAME => 'testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 04:00:18,240 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:18,240 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:18,240 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:18,241 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:18,242 DEBUG [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf 2023-02-11 04:00:18,242 DEBUG [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf 2023-02-11 04:00:18,242 INFO [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:18,242 INFO [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 747a6fa324aa972cea6bbd307f3b1539 columnFamilyName cf 2023-02-11 04:00:18,243 DEBUG [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf 2023-02-11 04:00:18,244 INFO [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] regionserver.HStore(310): Store=747a6fa324aa972cea6bbd307f3b1539/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:18,244 DEBUG [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf 2023-02-11 04:00:18,245 INFO [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 51e4d6a6293430d21cdf75126c54f808 columnFamilyName cf 2023-02-11 04:00:18,245 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:18,246 INFO [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] regionserver.HStore(310): Store=51e4d6a6293430d21cdf75126c54f808/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:18,247 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:18,248 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:18,249 DEBUG [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf 2023-02-11 04:00:18,249 DEBUG [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf 2023-02-11 04:00:18,250 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:18,250 INFO [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e1d0ef015d92b032f67c5052a11865c4 columnFamilyName cf 2023-02-11 04:00:18,252 INFO [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] regionserver.HStore(310): Store=e1d0ef015d92b032f67c5052a11865c4/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:18,253 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:18,254 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:18,255 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 747a6fa324aa972cea6bbd307f3b1539; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64898231, jitterRate=-0.032940998673439026}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:18,255 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 747a6fa324aa972cea6bbd307f3b1539: Running coprocessor pre-open hook at 1676088018238Writing region info on filesystem at 1676088018238Initializing all the Stores at 1676088018239 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088018240 (+1 ms)Cleaning up temporary data from old regions at 1676088018248 (+8 ms)Cleaning up detritus from prior splits at 1676088018248Running coprocessor post-open hooks at 1676088018255 (+7 ms)Region opened successfully at 1676088018255 2023-02-11 04:00:18,255 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:18,256 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:18,257 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539., pid=46, masterSystemTime=1676088018232 2023-02-11 04:00:18,257 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 51e4d6a6293430d21cdf75126c54f808; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70534125, jitterRate=0.05104036629199982}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:18,257 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 51e4d6a6293430d21cdf75126c54f808: Running coprocessor pre-open hook at 1676088018241Writing region info on filesystem at 1676088018241Initializing all the Stores at 1676088018242 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088018242Cleaning up temporary data from old regions at 1676088018250 (+8 ms)Cleaning up detritus from prior splits at 1676088018251 (+1 ms)Running coprocessor post-open hooks at 1676088018257 (+6 ms)Region opened successfully at 1676088018257 2023-02-11 04:00:18,259 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808., pid=47, masterSystemTime=1676088018234 2023-02-11 04:00:18,259 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:18,259 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:18,259 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:18,260 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d586dffdbc612899aa5c72d99bdc6519, NAME => 'testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 04:00:18,260 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:18,260 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:18,260 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=747a6fa324aa972cea6bbd307f3b1539, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:18,260 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:18,260 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088018260"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088018260"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088018260"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088018260"}]},"ts":"1676088018260"} 2023-02-11 04:00:18,261 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:18,260 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:18,262 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:18,262 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:18,262 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:18,262 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => aaa51003a74d8671fa0840bbfcb549ac, NAME => 'testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 04:00:18,262 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=39 updating hbase:meta row=51e4d6a6293430d21cdf75126c54f808, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:18,262 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:18,263 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088018262"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088018262"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088018262"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088018262"}]},"ts":"1676088018262"} 2023-02-11 04:00:18,263 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e1d0ef015d92b032f67c5052a11865c4; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71055194, jitterRate=0.05880489945411682}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:18,263 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:18,263 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e1d0ef015d92b032f67c5052a11865c4: Running coprocessor pre-open hook at 1676088018238Writing region info on filesystem at 1676088018238Initializing all the Stores at 1676088018239 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088018239Cleaning up temporary data from old regions at 1676088018256 (+17 ms)Cleaning up detritus from prior splits at 1676088018257 (+1 ms)Running coprocessor post-open hooks at 1676088018263 (+6 ms)Region opened successfully at 1676088018263 2023-02-11 04:00:18,263 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:18,263 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:18,264 INFO [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:18,264 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4., pid=48, masterSystemTime=1676088018233 2023-02-11 04:00:18,266 DEBUG [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf 2023-02-11 04:00:18,266 DEBUG [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf 2023-02-11 04:00:18,266 INFO [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:18,266 INFO [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d586dffdbc612899aa5c72d99bdc6519 columnFamilyName cf 2023-02-11 04:00:18,267 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:18,267 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:18,267 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:18,267 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 3452669615c81575c52ff96406a41e9e, NAME => 'testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 04:00:18,267 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:18,267 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:18,268 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:18,268 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:18,268 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=46, resume processing ppid=37 2023-02-11 04:00:18,268 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=46, ppid=37, state=SUCCESS; OpenRegionProcedure 747a6fa324aa972cea6bbd307f3b1539, server=jenkins-hbase4.apache.org,42385,1676088013771 in 180 msec 2023-02-11 04:00:18,268 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=e1d0ef015d92b032f67c5052a11865c4, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:18,268 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088018268"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088018268"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088018268"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088018268"}]},"ts":"1676088018268"} 2023-02-11 04:00:18,268 INFO [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] regionserver.HStore(310): Store=d586dffdbc612899aa5c72d99bdc6519/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:18,269 INFO [StoreOpener-3452669615c81575c52ff96406a41e9e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:18,270 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=47, resume processing ppid=39 2023-02-11 04:00:18,270 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=47, ppid=39, state=SUCCESS; OpenRegionProcedure 51e4d6a6293430d21cdf75126c54f808, server=jenkins-hbase4.apache.org,46851,1676088013694 in 181 msec 2023-02-11 04:00:18,270 DEBUG [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf 2023-02-11 04:00:18,270 DEBUG [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf 2023-02-11 04:00:18,271 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:18,271 INFO [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region aaa51003a74d8671fa0840bbfcb549ac columnFamilyName cf 2023-02-11 04:00:18,271 DEBUG [StoreOpener-3452669615c81575c52ff96406a41e9e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf 2023-02-11 04:00:18,272 DEBUG [StoreOpener-3452669615c81575c52ff96406a41e9e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf 2023-02-11 04:00:18,271 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, ASSIGN in 349 msec 2023-02-11 04:00:18,272 INFO [StoreOpener-3452669615c81575c52ff96406a41e9e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3452669615c81575c52ff96406a41e9e columnFamilyName cf 2023-02-11 04:00:18,273 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:18,273 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, ASSIGN in 351 msec 2023-02-11 04:00:18,274 INFO [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] regionserver.HStore(310): Store=aaa51003a74d8671fa0840bbfcb549ac/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:18,274 INFO [StoreOpener-3452669615c81575c52ff96406a41e9e-1] regionserver.HStore(310): Store=3452669615c81575c52ff96406a41e9e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:18,276 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:18,276 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=40 2023-02-11 04:00:18,277 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=40, state=SUCCESS; OpenRegionProcedure e1d0ef015d92b032f67c5052a11865c4, server=jenkins-hbase4.apache.org,40829,1676088013739 in 185 msec 2023-02-11 04:00:18,277 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:18,278 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:18,279 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, ASSIGN in 358 msec 2023-02-11 04:00:18,279 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:18,280 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:18,281 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened d586dffdbc612899aa5c72d99bdc6519; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75394804, jitterRate=0.12347012758255005}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:18,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for d586dffdbc612899aa5c72d99bdc6519: Running coprocessor pre-open hook at 1676088018261Writing region info on filesystem at 1676088018261Initializing all the Stores at 1676088018263 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088018263Cleaning up temporary data from old regions at 1676088018274 (+11 ms)Cleaning up detritus from prior splits at 1676088018275 (+1 ms)Running coprocessor post-open hooks at 1676088018281 (+6 ms)Region opened successfully at 1676088018281 2023-02-11 04:00:18,282 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519., pid=43, masterSystemTime=1676088018232 2023-02-11 04:00:18,284 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:18,284 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:18,284 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:18,285 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=d586dffdbc612899aa5c72d99bdc6519, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:18,285 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088018285"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088018285"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088018285"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088018285"}]},"ts":"1676088018285"} 2023-02-11 04:00:18,286 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 3452669615c81575c52ff96406a41e9e; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61354120, jitterRate=-0.08575236797332764}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:18,286 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 3452669615c81575c52ff96406a41e9e: Running coprocessor pre-open hook at 1676088018268Writing region info on filesystem at 1676088018268Initializing all the Stores at 1676088018269 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088018269Cleaning up temporary data from old regions at 1676088018279 (+10 ms)Cleaning up detritus from prior splits at 1676088018279Running coprocessor post-open hooks at 1676088018286 (+7 ms)Region opened successfully at 1676088018286 2023-02-11 04:00:18,286 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:18,287 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e., pid=44, masterSystemTime=1676088018233 2023-02-11 04:00:18,288 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened aaa51003a74d8671fa0840bbfcb549ac; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61621810, jitterRate=-0.08176347613334656}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:18,288 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for aaa51003a74d8671fa0840bbfcb549ac: Running coprocessor pre-open hook at 1676088018263Writing region info on filesystem at 1676088018263Initializing all the Stores at 1676088018265 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088018265Cleaning up temporary data from old regions at 1676088018280 (+15 ms)Cleaning up detritus from prior splits at 1676088018281 (+1 ms)Running coprocessor post-open hooks at 1676088018288 (+7 ms)Region opened successfully at 1676088018288 2023-02-11 04:00:18,289 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:18,289 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:18,289 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac., pid=45, masterSystemTime=1676088018234 2023-02-11 04:00:18,290 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=38 updating hbase:meta row=3452669615c81575c52ff96406a41e9e, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:18,290 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088018290"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088018290"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088018290"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088018290"}]},"ts":"1676088018290"} 2023-02-11 04:00:18,291 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=43, resume processing ppid=42 2023-02-11 04:00:18,291 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=43, ppid=42, state=SUCCESS; OpenRegionProcedure d586dffdbc612899aa5c72d99bdc6519, server=jenkins-hbase4.apache.org,42385,1676088013771 in 209 msec 2023-02-11 04:00:18,292 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:18,292 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:18,293 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=41 updating hbase:meta row=aaa51003a74d8671fa0840bbfcb549ac, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:18,293 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088018293"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088018293"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088018293"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088018293"}]},"ts":"1676088018293"} 2023-02-11 04:00:18,295 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, ASSIGN in 372 msec 2023-02-11 04:00:18,297 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=38 2023-02-11 04:00:18,297 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=38, state=SUCCESS; OpenRegionProcedure 3452669615c81575c52ff96406a41e9e, server=jenkins-hbase4.apache.org,40829,1676088013739 in 212 msec 2023-02-11 04:00:18,299 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=41 2023-02-11 04:00:18,299 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=41, state=SUCCESS; OpenRegionProcedure aaa51003a74d8671fa0840bbfcb549ac, server=jenkins-hbase4.apache.org,46851,1676088013694 in 213 msec 2023-02-11 04:00:18,300 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=38, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, ASSIGN in 378 msec 2023-02-11 04:00:18,302 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=36 2023-02-11 04:00:18,302 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, ASSIGN in 380 msec 2023-02-11 04:00:18,303 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088018302"}]},"ts":"1676088018302"} 2023-02-11 04:00:18,304 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676088016466, state=ENABLED in hbase:meta 2023-02-11 04:00:18,306 INFO [PEWorker-2] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676088016466' was successfully enabled. 2023-02-11 04:00:18,309 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=36, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676088016466 in 402 msec 2023-02-11 04:00:18,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-11 04:00:18,667 INFO [Listener at localhost/33753] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676088016466, procId: 36 completed 2023-02-11 04:00:18,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 04:00:18,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 04:00:18,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 04:00:18,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 04:00:18,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 04:00:18,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 04:00:18,722 DEBUG [Listener at localhost/33753] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676088016466 2023-02-11 04:00:18,722 DEBUG [Listener at localhost/33753] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:18,724 INFO [Listener at localhost/33753] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-11 04:00:18,745 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 04:00:18,747 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 04:00:18,750 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 04:00:18,752 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 04:00:18,753 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 04:00:18,764 INFO [Listener at localhost/33753] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676088016466 2023-02-11 04:00:18,764 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676088016466 2023-02-11 04:00:18,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] procedure2.ProcedureExecutor(1029): Stored pid=49, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:18,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-11 04:00:18,769 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088018769"}]},"ts":"1676088018769"} 2023-02-11 04:00:18,771 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676088016466, state=DISABLING in hbase:meta 2023-02-11 04:00:18,774 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676088016466 to state=DISABLING 2023-02-11 04:00:18,775 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, UNASSIGN}, {pid=51, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, UNASSIGN}, {pid=52, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, UNASSIGN}, {pid=53, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, UNASSIGN}, {pid=54, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, UNASSIGN}, {pid=55, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, UNASSIGN}] 2023-02-11 04:00:18,778 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, UNASSIGN 2023-02-11 04:00:18,778 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, UNASSIGN 2023-02-11 04:00:18,778 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=53, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, UNASSIGN 2023-02-11 04:00:18,778 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=54, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, UNASSIGN 2023-02-11 04:00:18,778 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, UNASSIGN 2023-02-11 04:00:18,779 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=747a6fa324aa972cea6bbd307f3b1539, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:18,779 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=3452669615c81575c52ff96406a41e9e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:18,779 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088018779"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088018779"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088018779"}]},"ts":"1676088018779"} 2023-02-11 04:00:18,780 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088018779"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088018779"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088018779"}]},"ts":"1676088018779"} 2023-02-11 04:00:18,780 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=53 updating hbase:meta row=e1d0ef015d92b032f67c5052a11865c4, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:18,780 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088018780"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088018780"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088018780"}]},"ts":"1676088018780"} 2023-02-11 04:00:18,781 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=aaa51003a74d8671fa0840bbfcb549ac, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:18,781 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=d586dffdbc612899aa5c72d99bdc6519, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:18,781 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088018780"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088018780"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088018780"}]},"ts":"1676088018780"} 2023-02-11 04:00:18,781 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088018781"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088018781"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088018781"}]},"ts":"1676088018781"} 2023-02-11 04:00:18,781 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=50, state=RUNNABLE; CloseRegionProcedure 747a6fa324aa972cea6bbd307f3b1539, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:18,782 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=57, ppid=51, state=RUNNABLE; CloseRegionProcedure 3452669615c81575c52ff96406a41e9e, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:18,784 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=53, state=RUNNABLE; CloseRegionProcedure e1d0ef015d92b032f67c5052a11865c4, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:18,785 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=54, state=RUNNABLE; CloseRegionProcedure aaa51003a74d8671fa0840bbfcb549ac, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:18,786 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=52, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, UNASSIGN 2023-02-11 04:00:18,787 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=55, state=RUNNABLE; CloseRegionProcedure d586dffdbc612899aa5c72d99bdc6519, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:18,788 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=51e4d6a6293430d21cdf75126c54f808, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:18,788 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088018788"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088018788"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088018788"}]},"ts":"1676088018788"} 2023-02-11 04:00:18,790 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=52, state=RUNNABLE; CloseRegionProcedure 51e4d6a6293430d21cdf75126c54f808, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:18,935 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:18,935 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing d586dffdbc612899aa5c72d99bdc6519, disabling compactions & flushes 2023-02-11 04:00:18,936 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:18,936 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:18,936 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. after waiting 0 ms 2023-02-11 04:00:18,936 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:18,936 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:18,937 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 3452669615c81575c52ff96406a41e9e, disabling compactions & flushes 2023-02-11 04:00:18,938 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:18,938 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:18,938 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. after waiting 0 ms 2023-02-11 04:00:18,938 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:18,938 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 3452669615c81575c52ff96406a41e9e 1/1 column families, dataSize=1.83 KB heapSize=4.19 KB 2023-02-11 04:00:18,938 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing d586dffdbc612899aa5c72d99bdc6519 1/1 column families, dataSize=22.96 KB heapSize=49.61 KB 2023-02-11 04:00:18,939 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:18,940 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing aaa51003a74d8671fa0840bbfcb549ac, disabling compactions & flushes 2023-02-11 04:00:18,940 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:18,940 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:18,940 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. after waiting 0 ms 2023-02-11 04:00:18,940 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:18,941 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing aaa51003a74d8671fa0840bbfcb549ac 1/1 column families, dataSize=1.70 KB heapSize=3.91 KB 2023-02-11 04:00:19,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-11 04:00:19,073 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 2023-02-11 04:00:19,073 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 2023-02-11 04:00:19,073 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 2023-02-11 04:00:19,123 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/.tmp/cf/3f27b1f3105b45bf81c17ba9c704bf44 2023-02-11 04:00:19,123 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/.tmp/cf/a11c828e49bd4e7a9cf05be9ff07ad7f 2023-02-11 04:00:19,125 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=23.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/.tmp/cf/e928d1188fee412e9c504ba31bbb3ef6 2023-02-11 04:00:19,135 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/.tmp/cf/3f27b1f3105b45bf81c17ba9c704bf44 as hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/3f27b1f3105b45bf81c17ba9c704bf44 2023-02-11 04:00:19,135 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/.tmp/cf/a11c828e49bd4e7a9cf05be9ff07ad7f as hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/a11c828e49bd4e7a9cf05be9ff07ad7f 2023-02-11 04:00:19,136 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/.tmp/cf/e928d1188fee412e9c504ba31bbb3ef6 as hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/e928d1188fee412e9c504ba31bbb3ef6 2023-02-11 04:00:19,144 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/e928d1188fee412e9c504ba31bbb3ef6, entries=351, sequenceid=8, filesize=64.7 K 2023-02-11 04:00:19,144 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/3f27b1f3105b45bf81c17ba9c704bf44, entries=26, sequenceid=8, filesize=9.3 K 2023-02-11 04:00:19,148 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/a11c828e49bd4e7a9cf05be9ff07ad7f, entries=28, sequenceid=8, filesize=9.7 K 2023-02-11 04:00:19,148 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~22.96 KB/23513, heapSize ~49.59 KB/50784, currentSize=0 B/0 for d586dffdbc612899aa5c72d99bdc6519 in 210ms, sequenceid=8, compaction requested=false 2023-02-11 04:00:19,148 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.70 KB/1740, heapSize ~3.89 KB/3984, currentSize=0 B/0 for aaa51003a74d8671fa0840bbfcb549ac in 208ms, sequenceid=8, compaction requested=false 2023-02-11 04:00:19,149 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.83 KB/1874, heapSize ~4.17 KB/4272, currentSize=0 B/0 for 3452669615c81575c52ff96406a41e9e in 211ms, sequenceid=8, compaction requested=false 2023-02-11 04:00:19,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676088016466' 2023-02-11 04:00:19,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676088016466' 2023-02-11 04:00:19,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676088016466' 2023-02-11 04:00:19,161 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 04:00:19,161 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 04:00:19,161 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 04:00:19,162 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:19,162 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 3452669615c81575c52ff96406a41e9e: Waiting for close lock at 1676088018936Running coprocessor pre-close hooks at 1676088018936Disabling compacts and flushes for region at 1676088018936Disabling writes for close at 1676088018938 (+2 ms)Obtaining lock to block concurrent updates at 1676088018938Preparing flush snapshotting stores in 3452669615c81575c52ff96406a41e9e at 1676088018938Finished memstore snapshotting testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e., syncing WAL and waiting on mvcc, flushsize=dataSize=1874, getHeapSize=4272, getOffHeapSize=0, getCellsCount=28 at 1676088018949 (+11 ms)Flushing stores of testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. at 1676088018950 (+1 ms)Flushing 3452669615c81575c52ff96406a41e9e/cf: creating writer at 1676088018953 (+3 ms)Flushing mob file 3452669615c81575c52ff96406a41e9e/cf: appending metadata at 1676088018996 (+43 ms)Flushing mob file 3452669615c81575c52ff96406a41e9e/cf: closing flushed file at 1676088018996Flushing 3452669615c81575c52ff96406a41e9e/cf: appending metadata at 1676088019076 (+80 ms)Flushing 3452669615c81575c52ff96406a41e9e/cf: closing flushed file at 1676088019079 (+3 ms)Flushing 3452669615c81575c52ff96406a41e9e/cf: reopening flushed file at 1676088019137 (+58 ms)Finished flush of dataSize ~1.83 KB/1874, heapSize ~4.17 KB/4272, currentSize=0 B/0 for 3452669615c81575c52ff96406a41e9e in 211ms, sequenceid=8, compaction requested=false at 1676088019149 (+12 ms)Writing region close event to WAL at 1676088019156 (+7 ms)Running coprocessor post-close hooks at 1676088019162 (+6 ms)Closed at 1676088019162 2023-02-11 04:00:19,162 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:19,162 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for aaa51003a74d8671fa0840bbfcb549ac: Waiting for close lock at 1676088018939Running coprocessor pre-close hooks at 1676088018939Disabling compacts and flushes for region at 1676088018939Disabling writes for close at 1676088018940 (+1 ms)Obtaining lock to block concurrent updates at 1676088018941 (+1 ms)Preparing flush snapshotting stores in aaa51003a74d8671fa0840bbfcb549ac at 1676088018941Finished memstore snapshotting testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac., syncing WAL and waiting on mvcc, flushsize=dataSize=1740, getHeapSize=3984, getOffHeapSize=0, getCellsCount=26 at 1676088018949 (+8 ms)Flushing stores of testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. at 1676088018950 (+1 ms)Flushing aaa51003a74d8671fa0840bbfcb549ac/cf: creating writer at 1676088018953 (+3 ms)Flushing mob file aaa51003a74d8671fa0840bbfcb549ac/cf: appending metadata at 1676088018996 (+43 ms)Flushing mob file aaa51003a74d8671fa0840bbfcb549ac/cf: closing flushed file at 1676088018996Flushing aaa51003a74d8671fa0840bbfcb549ac/cf: appending metadata at 1676088019075 (+79 ms)Flushing aaa51003a74d8671fa0840bbfcb549ac/cf: closing flushed file at 1676088019079 (+4 ms)Flushing aaa51003a74d8671fa0840bbfcb549ac/cf: reopening flushed file at 1676088019137 (+58 ms)Finished flush of dataSize ~1.70 KB/1740, heapSize ~3.89 KB/3984, currentSize=0 B/0 for aaa51003a74d8671fa0840bbfcb549ac in 208ms, sequenceid=8, compaction requested=false at 1676088019148 (+11 ms)Writing region close event to WAL at 1676088019156 (+8 ms)Running coprocessor post-close hooks at 1676088019162 (+6 ms)Closed at 1676088019162 2023-02-11 04:00:19,162 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:19,162 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for d586dffdbc612899aa5c72d99bdc6519: Waiting for close lock at 1676088018935Running coprocessor pre-close hooks at 1676088018935Disabling compacts and flushes for region at 1676088018935Disabling writes for close at 1676088018936 (+1 ms)Obtaining lock to block concurrent updates at 1676088018938 (+2 ms)Preparing flush snapshotting stores in d586dffdbc612899aa5c72d99bdc6519 at 1676088018938Finished memstore snapshotting testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519., syncing WAL and waiting on mvcc, flushsize=dataSize=23513, getHeapSize=50784, getOffHeapSize=0, getCellsCount=351 at 1676088018949 (+11 ms)Flushing stores of testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. at 1676088018950 (+1 ms)Flushing d586dffdbc612899aa5c72d99bdc6519/cf: creating writer at 1676088018953 (+3 ms)Flushing mob file d586dffdbc612899aa5c72d99bdc6519/cf: appending metadata at 1676088019013 (+60 ms)Flushing mob file d586dffdbc612899aa5c72d99bdc6519/cf: closing flushed file at 1676088019013Flushing d586dffdbc612899aa5c72d99bdc6519/cf: appending metadata at 1676088019076 (+63 ms)Flushing d586dffdbc612899aa5c72d99bdc6519/cf: closing flushed file at 1676088019079 (+3 ms)Flushing d586dffdbc612899aa5c72d99bdc6519/cf: reopening flushed file at 1676088019137 (+58 ms)Finished flush of dataSize ~22.96 KB/23513, heapSize ~49.59 KB/50784, currentSize=0 B/0 for d586dffdbc612899aa5c72d99bdc6519 in 210ms, sequenceid=8, compaction requested=false at 1676088019148 (+11 ms)Writing region close event to WAL at 1676088019156 (+8 ms)Running coprocessor post-close hooks at 1676088019162 (+6 ms)Closed at 1676088019162 2023-02-11 04:00:19,165 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:19,165 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:19,165 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e1d0ef015d92b032f67c5052a11865c4, disabling compactions & flushes 2023-02-11 04:00:19,166 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:19,166 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:19,166 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. after waiting 0 ms 2023-02-11 04:00:19,166 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:19,166 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing e1d0ef015d92b032f67c5052a11865c4 1/1 column families, dataSize=2.29 KB heapSize=5.17 KB 2023-02-11 04:00:19,167 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=3452669615c81575c52ff96406a41e9e, regionState=CLOSED 2023-02-11 04:00:19,167 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088019166"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088019166"}]},"ts":"1676088019166"} 2023-02-11 04:00:19,167 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:19,168 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:19,168 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 51e4d6a6293430d21cdf75126c54f808, disabling compactions & flushes 2023-02-11 04:00:19,169 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:19,169 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:19,169 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. after waiting 0 ms 2023-02-11 04:00:19,169 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=aaa51003a74d8671fa0840bbfcb549ac, regionState=CLOSED 2023-02-11 04:00:19,169 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:19,170 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088019169"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088019169"}]},"ts":"1676088019169"} 2023-02-11 04:00:19,170 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:19,170 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:19,170 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 747a6fa324aa972cea6bbd307f3b1539, disabling compactions & flushes 2023-02-11 04:00:19,170 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:19,170 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:19,170 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. after waiting 0 ms 2023-02-11 04:00:19,170 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:19,170 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 51e4d6a6293430d21cdf75126c54f808 1/1 column families, dataSize=2.42 KB heapSize=5.45 KB 2023-02-11 04:00:19,170 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 747a6fa324aa972cea6bbd307f3b1539 1/1 column families, dataSize=1.50 KB heapSize=3.48 KB 2023-02-11 04:00:19,173 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=d586dffdbc612899aa5c72d99bdc6519, regionState=CLOSED 2023-02-11 04:00:19,173 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088019172"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088019172"}]},"ts":"1676088019172"} 2023-02-11 04:00:19,177 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=57, resume processing ppid=51 2023-02-11 04:00:19,178 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=57, ppid=51, state=SUCCESS; CloseRegionProcedure 3452669615c81575c52ff96406a41e9e, server=jenkins-hbase4.apache.org,40829,1676088013739 in 389 msec 2023-02-11 04:00:19,179 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=54 2023-02-11 04:00:19,179 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=54, state=SUCCESS; CloseRegionProcedure aaa51003a74d8671fa0840bbfcb549ac, server=jenkins-hbase4.apache.org,46851,1676088013694 in 389 msec 2023-02-11 04:00:19,181 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=55 2023-02-11 04:00:19,181 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, UNASSIGN in 403 msec 2023-02-11 04:00:19,181 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=55, state=SUCCESS; CloseRegionProcedure d586dffdbc612899aa5c72d99bdc6519, server=jenkins-hbase4.apache.org,42385,1676088013771 in 389 msec 2023-02-11 04:00:19,182 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=54, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, UNASSIGN in 404 msec 2023-02-11 04:00:19,182 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, UNASSIGN in 406 msec 2023-02-11 04:00:19,227 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 2023-02-11 04:00:19,231 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 2023-02-11 04:00:19,238 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 2023-02-11 04:00:19,248 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/.tmp/cf/93d3b4e383ae4310a66075c83ee5c664 2023-02-11 04:00:19,257 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/.tmp/cf/93d3b4e383ae4310a66075c83ee5c664 as hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/93d3b4e383ae4310a66075c83ee5c664 2023-02-11 04:00:19,262 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/.tmp/cf/1b38064ae3304eca9c176d5669359823 2023-02-11 04:00:19,263 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/.tmp/cf/0a646c34d29c4963afc688a0da1fd6d4 2023-02-11 04:00:19,267 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/93d3b4e383ae4310a66075c83ee5c664, entries=35, sequenceid=8, filesize=10.9 K 2023-02-11 04:00:19,270 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.29 KB/2343, heapSize ~5.16 KB/5280, currentSize=0 B/0 for e1d0ef015d92b032f67c5052a11865c4 in 104ms, sequenceid=8, compaction requested=false 2023-02-11 04:00:19,274 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/.tmp/cf/0a646c34d29c4963afc688a0da1fd6d4 as hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/0a646c34d29c4963afc688a0da1fd6d4 2023-02-11 04:00:19,274 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/.tmp/cf/1b38064ae3304eca9c176d5669359823 as hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/1b38064ae3304eca9c176d5669359823 2023-02-11 04:00:19,279 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 04:00:19,280 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:19,280 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e1d0ef015d92b032f67c5052a11865c4: Waiting for close lock at 1676088019165Running coprocessor pre-close hooks at 1676088019165Disabling compacts and flushes for region at 1676088019165Disabling writes for close at 1676088019166 (+1 ms)Obtaining lock to block concurrent updates at 1676088019166Preparing flush snapshotting stores in e1d0ef015d92b032f67c5052a11865c4 at 1676088019166Finished memstore snapshotting testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4., syncing WAL and waiting on mvcc, flushsize=dataSize=2343, getHeapSize=5280, getOffHeapSize=0, getCellsCount=35 at 1676088019166Flushing stores of testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. at 1676088019169 (+3 ms)Flushing e1d0ef015d92b032f67c5052a11865c4/cf: creating writer at 1676088019169Flushing mob file e1d0ef015d92b032f67c5052a11865c4/cf: appending metadata at 1676088019185 (+16 ms)Flushing mob file e1d0ef015d92b032f67c5052a11865c4/cf: closing flushed file at 1676088019185Flushing e1d0ef015d92b032f67c5052a11865c4/cf: appending metadata at 1676088019228 (+43 ms)Flushing e1d0ef015d92b032f67c5052a11865c4/cf: closing flushed file at 1676088019228Flushing e1d0ef015d92b032f67c5052a11865c4/cf: reopening flushed file at 1676088019260 (+32 ms)Finished flush of dataSize ~2.29 KB/2343, heapSize ~5.16 KB/5280, currentSize=0 B/0 for e1d0ef015d92b032f67c5052a11865c4 in 104ms, sequenceid=8, compaction requested=false at 1676088019270 (+10 ms)Writing region close event to WAL at 1676088019272 (+2 ms)Running coprocessor post-close hooks at 1676088019280 (+8 ms)Closed at 1676088019280 2023-02-11 04:00:19,282 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/1b38064ae3304eca9c176d5669359823, entries=37, sequenceid=8, filesize=11.2 K 2023-02-11 04:00:19,283 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.42 KB/2477, heapSize ~5.44 KB/5568, currentSize=0 B/0 for 51e4d6a6293430d21cdf75126c54f808 in 113ms, sequenceid=8, compaction requested=false 2023-02-11 04:00:19,283 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=53 updating hbase:meta row=e1d0ef015d92b032f67c5052a11865c4, regionState=CLOSED 2023-02-11 04:00:19,284 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088019283"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088019283"}]},"ts":"1676088019283"} 2023-02-11 04:00:19,285 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:19,289 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/0a646c34d29c4963afc688a0da1fd6d4, entries=23, sequenceid=8, filesize=8.8 K 2023-02-11 04:00:19,290 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.50 KB/1539, heapSize ~3.47 KB/3552, currentSize=0 B/0 for 747a6fa324aa972cea6bbd307f3b1539 in 120ms, sequenceid=8, compaction requested=false 2023-02-11 04:00:19,290 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 04:00:19,292 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=53 2023-02-11 04:00:19,292 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=53, state=SUCCESS; CloseRegionProcedure e1d0ef015d92b032f67c5052a11865c4, server=jenkins-hbase4.apache.org,40829,1676088013739 in 504 msec 2023-02-11 04:00:19,293 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:19,293 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 51e4d6a6293430d21cdf75126c54f808: Waiting for close lock at 1676088019168Running coprocessor pre-close hooks at 1676088019168Disabling compacts and flushes for region at 1676088019168Disabling writes for close at 1676088019169 (+1 ms)Obtaining lock to block concurrent updates at 1676088019170 (+1 ms)Preparing flush snapshotting stores in 51e4d6a6293430d21cdf75126c54f808 at 1676088019170Finished memstore snapshotting testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808., syncing WAL and waiting on mvcc, flushsize=dataSize=2477, getHeapSize=5568, getOffHeapSize=0, getCellsCount=37 at 1676088019171 (+1 ms)Flushing stores of testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. at 1676088019172 (+1 ms)Flushing 51e4d6a6293430d21cdf75126c54f808/cf: creating writer at 1676088019172Flushing mob file 51e4d6a6293430d21cdf75126c54f808/cf: appending metadata at 1676088019190 (+18 ms)Flushing mob file 51e4d6a6293430d21cdf75126c54f808/cf: closing flushed file at 1676088019190Flushing 51e4d6a6293430d21cdf75126c54f808/cf: appending metadata at 1676088019233 (+43 ms)Flushing 51e4d6a6293430d21cdf75126c54f808/cf: closing flushed file at 1676088019233Flushing 51e4d6a6293430d21cdf75126c54f808/cf: reopening flushed file at 1676088019275 (+42 ms)Finished flush of dataSize ~2.42 KB/2477, heapSize ~5.44 KB/5568, currentSize=0 B/0 for 51e4d6a6293430d21cdf75126c54f808 in 113ms, sequenceid=8, compaction requested=false at 1676088019283 (+8 ms)Writing region close event to WAL at 1676088019285 (+2 ms)Running coprocessor post-close hooks at 1676088019293 (+8 ms)Closed at 1676088019293 2023-02-11 04:00:19,295 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=53, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, UNASSIGN in 517 msec 2023-02-11 04:00:19,295 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:19,296 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=51e4d6a6293430d21cdf75126c54f808, regionState=CLOSED 2023-02-11 04:00:19,296 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088019296"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088019296"}]},"ts":"1676088019296"} 2023-02-11 04:00:19,296 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 04:00:19,297 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:19,297 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 747a6fa324aa972cea6bbd307f3b1539: Waiting for close lock at 1676088019170Running coprocessor pre-close hooks at 1676088019170Disabling compacts and flushes for region at 1676088019170Disabling writes for close at 1676088019170Obtaining lock to block concurrent updates at 1676088019171 (+1 ms)Preparing flush snapshotting stores in 747a6fa324aa972cea6bbd307f3b1539 at 1676088019171Finished memstore snapshotting testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539., syncing WAL and waiting on mvcc, flushsize=dataSize=1539, getHeapSize=3552, getOffHeapSize=0, getCellsCount=23 at 1676088019171Flushing stores of testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. at 1676088019172 (+1 ms)Flushing 747a6fa324aa972cea6bbd307f3b1539/cf: creating writer at 1676088019172Flushing mob file 747a6fa324aa972cea6bbd307f3b1539/cf: appending metadata at 1676088019190 (+18 ms)Flushing mob file 747a6fa324aa972cea6bbd307f3b1539/cf: closing flushed file at 1676088019190Flushing 747a6fa324aa972cea6bbd307f3b1539/cf: appending metadata at 1676088019239 (+49 ms)Flushing 747a6fa324aa972cea6bbd307f3b1539/cf: closing flushed file at 1676088019239Flushing 747a6fa324aa972cea6bbd307f3b1539/cf: reopening flushed file at 1676088019275 (+36 ms)Finished flush of dataSize ~1.50 KB/1539, heapSize ~3.47 KB/3552, currentSize=0 B/0 for 747a6fa324aa972cea6bbd307f3b1539 in 120ms, sequenceid=8, compaction requested=false at 1676088019290 (+15 ms)Writing region close event to WAL at 1676088019292 (+2 ms)Running coprocessor post-close hooks at 1676088019297 (+5 ms)Closed at 1676088019297 2023-02-11 04:00:19,299 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:19,300 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=747a6fa324aa972cea6bbd307f3b1539, regionState=CLOSED 2023-02-11 04:00:19,300 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088019300"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088019300"}]},"ts":"1676088019300"} 2023-02-11 04:00:19,301 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=52 2023-02-11 04:00:19,301 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=52, state=SUCCESS; CloseRegionProcedure 51e4d6a6293430d21cdf75126c54f808, server=jenkins-hbase4.apache.org,46851,1676088013694 in 508 msec 2023-02-11 04:00:19,303 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, UNASSIGN in 526 msec 2023-02-11 04:00:19,305 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=50 2023-02-11 04:00:19,305 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=50, state=SUCCESS; CloseRegionProcedure 747a6fa324aa972cea6bbd307f3b1539, server=jenkins-hbase4.apache.org,42385,1676088013771 in 521 msec 2023-02-11 04:00:19,310 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=50, resume processing ppid=49 2023-02-11 04:00:19,310 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=50, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, UNASSIGN in 530 msec 2023-02-11 04:00:19,311 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088019311"}]},"ts":"1676088019311"} 2023-02-11 04:00:19,313 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676088016466, state=DISABLED in hbase:meta 2023-02-11 04:00:19,315 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676088016466 to state=DISABLED 2023-02-11 04:00:19,318 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=49, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676088016466 in 552 msec 2023-02-11 04:00:19,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-11 04:00:19,524 INFO [Listener at localhost/33753] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676088016466, procId: 49 completed 2023-02-11 04:00:19,525 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=snaptb0-1676088016466 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 } 2023-02-11 04:00:19,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676088019525 (current time:1676088019525). 2023-02-11 04:00:19,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-11 04:00:19,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] zookeeper.ReadOnlyZKClient(139): Connect 0x1f67ed1e to 127.0.0.1:63425 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 04:00:19,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6b87fb4e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 04:00:19,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 04:00:19,534 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55148, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 04:00:19,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1f67ed1e to 127.0.0.1:63425 2023-02-11 04:00:19,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:00:19,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-11 04:00:19,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb0-1676088016466 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 } 2023-02-11 04:00:19,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676088016466, type=EXCLUSIVE 2023-02-11 04:00:19,540 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=62, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676088016466, type=EXCLUSIVE 2023-02-11 04:00:19,540 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=62, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676088016466, type=EXCLUSIVE; timeout=600000, timestamp=1676088619540 2023-02-11 04:00:19,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb0-1676088016466 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 } 2023-02-11 04:00:19,540 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb0-1676088016466 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676088016466 2023-02-11 04:00:19,541 DEBUG [Listener at localhost/33753] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb0-1676088016466 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-11 04:00:19,541 DEBUG [Listener at localhost/33753] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-11 04:00:19,560 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676088016466 table=testRestoreSnapshot-1676088016466 type=DISABLED ttl=0 } 2023-02-11 04:00:19,562 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:19,562 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:19,562 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:19,562 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:19,562 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:19,563 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:19,563 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:19,563 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:19,563 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:19,563 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:19,563 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:19,563 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:19,563 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:19,563 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:19,566 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/93d3b4e383ae4310a66075c83ee5c664] hfiles 2023-02-11 04:00:19,566 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/93d3b4e383ae4310a66075c83ee5c664 2023-02-11 04:00:19,566 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/e928d1188fee412e9c504ba31bbb3ef6] hfiles 2023-02-11 04:00:19,566 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/e928d1188fee412e9c504ba31bbb3ef6 2023-02-11 04:00:19,567 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/1b38064ae3304eca9c176d5669359823] hfiles 2023-02-11 04:00:19,567 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/1b38064ae3304eca9c176d5669359823 2023-02-11 04:00:19,567 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/0a646c34d29c4963afc688a0da1fd6d4] hfiles 2023-02-11 04:00:19,567 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/0a646c34d29c4963afc688a0da1fd6d4 2023-02-11 04:00:19,569 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/3f27b1f3105b45bf81c17ba9c704bf44] hfiles 2023-02-11 04:00:19,569 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/3f27b1f3105b45bf81c17ba9c704bf44 2023-02-11 04:00:19,573 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/a11c828e49bd4e7a9cf05be9ff07ad7f] hfiles 2023-02-11 04:00:19,573 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/a11c828e49bd4e7a9cf05be9ff07ad7f 2023-02-11 04:00:19,574 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45] hfiles 2023-02-11 04:00:19,575 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 2023-02-11 04:00:19,575 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 2023-02-11 04:00:19,575 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 2023-02-11 04:00:19,575 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 2023-02-11 04:00:19,575 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 2023-02-11 04:00:19,575 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 2023-02-11 04:00:19,633 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb0-1676088016466 table=testRestoreSnapshot-1676088016466 type=DISABLED ttl=0 } as finished. 2023-02-11 04:00:19,633 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb0-1676088016466 2023-02-11 04:00:19,634 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/.tmp/snaptb0-1676088016466 2023-02-11 04:00:19,676 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/.tmp/snaptb0-1676088016466 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/snaptb0-1676088016466 2023-02-11 04:00:19,702 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb0-1676088016466 of table testRestoreSnapshot-1676088016466 completed 2023-02-11 04:00:19,702 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/.tmp/snaptb0-1676088016466 2023-02-11 04:00:19,703 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/.tmp/snaptb0-1676088016466 2023-02-11 04:00:19,703 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb0-1676088016466 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676088016466 at 1676088019541Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676088016466 table=testRestoreSnapshot-1676088016466 type=DISABLED ttl=0 } at 1676088019560 (+19 ms)Consolidate snapshot: snaptb0-1676088016466 at 1676088019633 (+73 ms)Loading Region manifests for snaptb0-1676088016466 at 1676088019634 (+1 ms)Writing data manifest for snaptb0-1676088016466 at 1676088019650 (+16 ms)Verifying snapshot: snaptb0-1676088016466 at 1676088019661 (+11 ms)Snapshot snaptb0-1676088016466 of table testRestoreSnapshot-1676088016466 completed at 1676088019702 (+41 ms) 2023-02-11 04:00:19,704 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=62, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676088016466, type=EXCLUSIVE 2023-02-11 04:00:19,706 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676088016466, type=EXCLUSIVE in 166 msec 2023-02-11 04:00:19,791 DEBUG [Listener at localhost/33753] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 04:00:19,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676088016466 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 } is done 2023-02-11 04:00:19,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb0-1676088016466 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-11 04:00:19,793 INFO [Listener at localhost/33753] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676088016466 2023-02-11 04:00:19,793 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676088016466 2023-02-11 04:00:19,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] procedure2.ProcedureExecutor(1029): Stored pid=63, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:19,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-11 04:00:19,798 INFO [PEWorker-5] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676088016466 2023-02-11 04:00:19,798 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088019798"}]},"ts":"1676088019798"} 2023-02-11 04:00:19,800 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676088016466, state=ENABLING in hbase:meta 2023-02-11 04:00:19,805 DEBUG [PEWorker-5] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-11 04:00:19,805 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, ASSIGN}, {pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, ASSIGN}, {pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, ASSIGN}, {pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, ASSIGN}, {pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, ASSIGN}, {pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, ASSIGN}] 2023-02-11 04:00:19,807 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, ASSIGN 2023-02-11 04:00:19,807 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, ASSIGN 2023-02-11 04:00:19,807 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, ASSIGN 2023-02-11 04:00:19,807 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, ASSIGN 2023-02-11 04:00:19,808 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:19,808 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:19,808 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,40829,1676088013739; forceNewPlan=false, retain=true 2023-02-11 04:00:19,808 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,40829,1676088013739; forceNewPlan=false, retain=true 2023-02-11 04:00:19,808 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:19,808 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, ASSIGN 2023-02-11 04:00:19,808 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:19,808 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,42385,1676088013771; forceNewPlan=false, retain=true 2023-02-11 04:00:19,809 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,46851,1676088013694; forceNewPlan=false, retain=true 2023-02-11 04:00:19,812 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, ASSIGN 2023-02-11 04:00:19,812 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:19,813 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,42385,1676088013771; forceNewPlan=false, retain=true 2023-02-11 04:00:19,813 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:19,813 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,46851,1676088013694; forceNewPlan=false, retain=true 2023-02-11 04:00:19,959 INFO [jenkins-hbase4:41713] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-11 04:00:19,959 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=68 updating hbase:meta row=aaa51003a74d8671fa0840bbfcb549ac, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:19,959 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=3452669615c81575c52ff96406a41e9e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:19,959 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088019959"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088019959"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088019959"}]},"ts":"1676088019959"} 2023-02-11 04:00:19,959 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=69 updating hbase:meta row=d586dffdbc612899aa5c72d99bdc6519, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:19,959 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=747a6fa324aa972cea6bbd307f3b1539, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:19,959 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=66 updating hbase:meta row=51e4d6a6293430d21cdf75126c54f808, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:19,959 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088019959"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088019959"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088019959"}]},"ts":"1676088019959"} 2023-02-11 04:00:19,959 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088019959"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088019959"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088019959"}]},"ts":"1676088019959"} 2023-02-11 04:00:19,959 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088019959"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088019959"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088019959"}]},"ts":"1676088019959"} 2023-02-11 04:00:19,959 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088019959"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088019959"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088019959"}]},"ts":"1676088019959"} 2023-02-11 04:00:19,961 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=70, ppid=68, state=RUNNABLE; OpenRegionProcedure aaa51003a74d8671fa0840bbfcb549ac, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:19,963 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=71, ppid=69, state=RUNNABLE; OpenRegionProcedure d586dffdbc612899aa5c72d99bdc6519, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:19,964 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=72, ppid=65, state=RUNNABLE; OpenRegionProcedure 3452669615c81575c52ff96406a41e9e, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:19,965 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=73, ppid=66, state=RUNNABLE; OpenRegionProcedure 51e4d6a6293430d21cdf75126c54f808, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:19,966 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=74, ppid=64, state=RUNNABLE; OpenRegionProcedure 747a6fa324aa972cea6bbd307f3b1539, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:19,968 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=67 updating hbase:meta row=e1d0ef015d92b032f67c5052a11865c4, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:19,968 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088019968"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088019968"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088019968"}]},"ts":"1676088019968"} 2023-02-11 04:00:19,972 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=75, ppid=67, state=RUNNABLE; OpenRegionProcedure e1d0ef015d92b032f67c5052a11865c4, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:20,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-11 04:00:20,120 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:20,120 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:20,120 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d586dffdbc612899aa5c72d99bdc6519, NAME => 'testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 04:00:20,120 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => aaa51003a74d8671fa0840bbfcb549ac, NAME => 'testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 04:00:20,120 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:20,120 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:20,120 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:20,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:20,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 3452669615c81575c52ff96406a41e9e, NAME => 'testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 04:00:20,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:20,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:20,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:20,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:20,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:20,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:20,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:20,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:20,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:20,122 INFO [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:20,123 INFO [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:20,123 INFO [StoreOpener-3452669615c81575c52ff96406a41e9e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:20,124 DEBUG [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf 2023-02-11 04:00:20,124 DEBUG [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf 2023-02-11 04:00:20,124 DEBUG [StoreOpener-3452669615c81575c52ff96406a41e9e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf 2023-02-11 04:00:20,124 DEBUG [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf 2023-02-11 04:00:20,124 DEBUG [StoreOpener-3452669615c81575c52ff96406a41e9e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf 2023-02-11 04:00:20,124 DEBUG [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf 2023-02-11 04:00:20,124 INFO [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region aaa51003a74d8671fa0840bbfcb549ac columnFamilyName cf 2023-02-11 04:00:20,125 INFO [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d586dffdbc612899aa5c72d99bdc6519 columnFamilyName cf 2023-02-11 04:00:20,125 INFO [StoreOpener-3452669615c81575c52ff96406a41e9e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3452669615c81575c52ff96406a41e9e columnFamilyName cf 2023-02-11 04:00:20,135 DEBUG [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/e928d1188fee412e9c504ba31bbb3ef6 2023-02-11 04:00:20,135 DEBUG [StoreOpener-3452669615c81575c52ff96406a41e9e-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/a11c828e49bd4e7a9cf05be9ff07ad7f 2023-02-11 04:00:20,136 DEBUG [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/3f27b1f3105b45bf81c17ba9c704bf44 2023-02-11 04:00:20,136 INFO [StoreOpener-3452669615c81575c52ff96406a41e9e-1] regionserver.HStore(310): Store=3452669615c81575c52ff96406a41e9e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:20,136 INFO [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] regionserver.HStore(310): Store=d586dffdbc612899aa5c72d99bdc6519/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:20,136 INFO [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] regionserver.HStore(310): Store=aaa51003a74d8671fa0840bbfcb549ac/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:20,138 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:20,139 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:20,139 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:20,140 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:20,141 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:20,141 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:20,144 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:20,145 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:20,145 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:20,146 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened aaa51003a74d8671fa0840bbfcb549ac; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67685146, jitterRate=0.008587270975112915}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:20,146 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for aaa51003a74d8671fa0840bbfcb549ac: Running coprocessor pre-open hook at 1676088020121Writing region info on filesystem at 1676088020121Initializing all the Stores at 1676088020122 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088020122Cleaning up temporary data from old regions at 1676088020140 (+18 ms)Cleaning up detritus from prior splits at 1676088020141 (+1 ms)Running coprocessor post-open hooks at 1676088020146 (+5 ms)Region opened successfully at 1676088020146 2023-02-11 04:00:20,146 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 3452669615c81575c52ff96406a41e9e; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70371870, jitterRate=0.048622578382492065}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:20,146 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened d586dffdbc612899aa5c72d99bdc6519; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64584226, jitterRate=-0.03762003779411316}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:20,146 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 3452669615c81575c52ff96406a41e9e: Running coprocessor pre-open hook at 1676088020121Writing region info on filesystem at 1676088020121Initializing all the Stores at 1676088020122 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088020122Cleaning up temporary data from old regions at 1676088020141 (+19 ms)Cleaning up detritus from prior splits at 1676088020142 (+1 ms)Running coprocessor post-open hooks at 1676088020146 (+4 ms)Region opened successfully at 1676088020146 2023-02-11 04:00:20,146 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for d586dffdbc612899aa5c72d99bdc6519: Running coprocessor pre-open hook at 1676088020121Writing region info on filesystem at 1676088020121Initializing all the Stores at 1676088020122 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088020122Cleaning up temporary data from old regions at 1676088020141 (+19 ms)Cleaning up detritus from prior splits at 1676088020142 (+1 ms)Running coprocessor post-open hooks at 1676088020146 (+4 ms)Region opened successfully at 1676088020146 2023-02-11 04:00:20,147 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac., pid=70, masterSystemTime=1676088020115 2023-02-11 04:00:20,148 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e., pid=72, masterSystemTime=1676088020116 2023-02-11 04:00:20,148 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519., pid=71, masterSystemTime=1676088020116 2023-02-11 04:00:20,150 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:20,150 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:20,150 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:20,150 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 51e4d6a6293430d21cdf75126c54f808, NAME => 'testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 04:00:20,150 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=68 updating hbase:meta row=aaa51003a74d8671fa0840bbfcb549ac, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:20,150 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:20,150 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:20,151 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088020150"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088020150"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088020150"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088020150"}]},"ts":"1676088020150"} 2023-02-11 04:00:20,151 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:20,151 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:20,151 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:20,151 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:20,151 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:20,152 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e1d0ef015d92b032f67c5052a11865c4, NAME => 'testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 04:00:20,152 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:20,152 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:20,152 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:20,152 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:20,152 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=3452669615c81575c52ff96406a41e9e, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:20,153 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088020152"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088020152"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088020152"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088020152"}]},"ts":"1676088020152"} 2023-02-11 04:00:20,154 INFO [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:20,154 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:20,154 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:20,154 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:20,154 INFO [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:20,154 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 747a6fa324aa972cea6bbd307f3b1539, NAME => 'testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 04:00:20,154 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:20,155 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:20,155 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:20,155 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:20,155 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=69 updating hbase:meta row=d586dffdbc612899aa5c72d99bdc6519, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:20,155 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088020155"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088020155"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088020155"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088020155"}]},"ts":"1676088020155"} 2023-02-11 04:00:20,157 DEBUG [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf 2023-02-11 04:00:20,157 DEBUG [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf 2023-02-11 04:00:20,157 DEBUG [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf 2023-02-11 04:00:20,157 DEBUG [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf 2023-02-11 04:00:20,157 INFO [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e1d0ef015d92b032f67c5052a11865c4 columnFamilyName cf 2023-02-11 04:00:20,158 INFO [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 51e4d6a6293430d21cdf75126c54f808 columnFamilyName cf 2023-02-11 04:00:20,159 INFO [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:20,159 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=70, resume processing ppid=68 2023-02-11 04:00:20,160 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=70, ppid=68, state=SUCCESS; OpenRegionProcedure aaa51003a74d8671fa0840bbfcb549ac, server=jenkins-hbase4.apache.org,46851,1676088013694 in 194 msec 2023-02-11 04:00:20,161 DEBUG [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf 2023-02-11 04:00:20,161 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=72, resume processing ppid=65 2023-02-11 04:00:20,161 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=72, ppid=65, state=SUCCESS; OpenRegionProcedure 3452669615c81575c52ff96406a41e9e, server=jenkins-hbase4.apache.org,40829,1676088013739 in 192 msec 2023-02-11 04:00:20,161 DEBUG [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf 2023-02-11 04:00:20,162 INFO [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 747a6fa324aa972cea6bbd307f3b1539 columnFamilyName cf 2023-02-11 04:00:20,164 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=71, resume processing ppid=69 2023-02-11 04:00:20,165 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=71, ppid=69, state=SUCCESS; OpenRegionProcedure d586dffdbc612899aa5c72d99bdc6519, server=jenkins-hbase4.apache.org,42385,1676088013771 in 195 msec 2023-02-11 04:00:20,165 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=68, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, ASSIGN in 355 msec 2023-02-11 04:00:20,165 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=65, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, ASSIGN in 356 msec 2023-02-11 04:00:20,167 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=69, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, ASSIGN in 360 msec 2023-02-11 04:00:20,169 DEBUG [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/93d3b4e383ae4310a66075c83ee5c664 2023-02-11 04:00:20,169 INFO [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] regionserver.HStore(310): Store=e1d0ef015d92b032f67c5052a11865c4/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:20,170 DEBUG [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/1b38064ae3304eca9c176d5669359823 2023-02-11 04:00:20,171 INFO [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] regionserver.HStore(310): Store=51e4d6a6293430d21cdf75126c54f808/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:20,171 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:20,172 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:20,173 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:20,174 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:20,174 DEBUG [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/0a646c34d29c4963afc688a0da1fd6d4 2023-02-11 04:00:20,174 INFO [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] regionserver.HStore(310): Store=747a6fa324aa972cea6bbd307f3b1539/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:20,175 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:20,177 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:20,177 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:20,178 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e1d0ef015d92b032f67c5052a11865c4; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62771982, jitterRate=-0.06462457776069641}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:20,178 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e1d0ef015d92b032f67c5052a11865c4: Running coprocessor pre-open hook at 1676088020152Writing region info on filesystem at 1676088020152Initializing all the Stores at 1676088020154 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088020154Cleaning up temporary data from old regions at 1676088020173 (+19 ms)Cleaning up detritus from prior splits at 1676088020174 (+1 ms)Running coprocessor post-open hooks at 1676088020178 (+4 ms)Region opened successfully at 1676088020178 2023-02-11 04:00:20,178 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:20,179 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4., pid=75, masterSystemTime=1676088020116 2023-02-11 04:00:20,180 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 51e4d6a6293430d21cdf75126c54f808; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58770417, jitterRate=-0.12425254285335541}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:20,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 51e4d6a6293430d21cdf75126c54f808: Running coprocessor pre-open hook at 1676088020151Writing region info on filesystem at 1676088020151Initializing all the Stores at 1676088020152 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088020152Cleaning up temporary data from old regions at 1676088020174 (+22 ms)Cleaning up detritus from prior splits at 1676088020175 (+1 ms)Running coprocessor post-open hooks at 1676088020180 (+5 ms)Region opened successfully at 1676088020180 2023-02-11 04:00:20,181 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808., pid=73, masterSystemTime=1676088020115 2023-02-11 04:00:20,182 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:20,182 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:20,182 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:20,182 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=67 updating hbase:meta row=e1d0ef015d92b032f67c5052a11865c4, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:20,183 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088020182"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088020182"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088020182"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088020182"}]},"ts":"1676088020182"} 2023-02-11 04:00:20,183 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 747a6fa324aa972cea6bbd307f3b1539; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75410803, jitterRate=0.12370853126049042}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:20,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 747a6fa324aa972cea6bbd307f3b1539: Running coprocessor pre-open hook at 1676088020155Writing region info on filesystem at 1676088020155Initializing all the Stores at 1676088020157 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088020157Cleaning up temporary data from old regions at 1676088020178 (+21 ms)Cleaning up detritus from prior splits at 1676088020179 (+1 ms)Running coprocessor post-open hooks at 1676088020183 (+4 ms)Region opened successfully at 1676088020183 2023-02-11 04:00:20,184 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539., pid=74, masterSystemTime=1676088020116 2023-02-11 04:00:20,184 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=66 updating hbase:meta row=51e4d6a6293430d21cdf75126c54f808, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:20,184 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:20,185 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088020184"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088020184"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088020184"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088020184"}]},"ts":"1676088020184"} 2023-02-11 04:00:20,185 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:20,187 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:20,187 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:20,188 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=747a6fa324aa972cea6bbd307f3b1539, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:20,188 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088020188"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088020188"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088020188"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088020188"}]},"ts":"1676088020188"} 2023-02-11 04:00:20,189 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=75, resume processing ppid=67 2023-02-11 04:00:20,189 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=75, ppid=67, state=SUCCESS; OpenRegionProcedure e1d0ef015d92b032f67c5052a11865c4, server=jenkins-hbase4.apache.org,40829,1676088013739 in 215 msec 2023-02-11 04:00:20,190 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=73, resume processing ppid=66 2023-02-11 04:00:20,190 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=73, ppid=66, state=SUCCESS; OpenRegionProcedure 51e4d6a6293430d21cdf75126c54f808, server=jenkins-hbase4.apache.org,46851,1676088013694 in 222 msec 2023-02-11 04:00:20,192 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=67, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, ASSIGN in 385 msec 2023-02-11 04:00:20,192 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=66, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, ASSIGN in 385 msec 2023-02-11 04:00:20,193 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=74, resume processing ppid=64 2023-02-11 04:00:20,193 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=74, ppid=64, state=SUCCESS; OpenRegionProcedure 747a6fa324aa972cea6bbd307f3b1539, server=jenkins-hbase4.apache.org,42385,1676088013771 in 224 msec 2023-02-11 04:00:20,195 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-02-11 04:00:20,195 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, ASSIGN in 388 msec 2023-02-11 04:00:20,196 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088020196"}]},"ts":"1676088020196"} 2023-02-11 04:00:20,197 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676088016466, state=ENABLED in hbase:meta 2023-02-11 04:00:20,199 INFO [PEWorker-3] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676088016466' was successfully enabled. 2023-02-11 04:00:20,201 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=63, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676088016466 in 406 msec 2023-02-11 04:00:20,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-11 04:00:20,550 INFO [Listener at localhost/33753] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676088016466, procId: 63 completed 2023-02-11 04:00:20,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 04:00:20,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 04:00:20,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 04:00:20,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 04:00:20,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 04:00:20,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 04:00:20,575 DEBUG [Listener at localhost/33753] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676088016466 2023-02-11 04:00:20,575 DEBUG [Listener at localhost/33753] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:20,577 INFO [Listener at localhost/33753] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-11 04:00:20,759 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 04:00:20,896 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 04:00:20,934 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-11 04:00:21,046 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-02-11 04:00:21,047 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-02-11 04:00:21,100 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 04:00:21,260 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 04:00:21,381 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 04:00:22,721 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-11 04:00:22,722 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-02-11 04:00:22,803 DEBUG [Listener at localhost/33753] zookeeper.ReadOnlyZKClient(139): Connect 0x490facbf to 127.0.0.1:63425 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 04:00:22,810 DEBUG [Listener at localhost/33753] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@290fa1b6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 04:00:22,813 DEBUG [hconnection-0x7b77b3f7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 04:00:22,816 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55156, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 04:00:22,821 DEBUG [hconnection-0x7b77b3f7-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 04:00:22,823 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49634, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 04:00:22,913 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 04:00:23,016 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 04:00:23,020 DEBUG [hconnection-0x7b77b3f7-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 04:00:23,022 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34728, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 04:00:23,152 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 04:00:23,287 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 04:00:23,383 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 04:00:24,599 INFO [Listener at localhost/33753] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676088016466 2023-02-11 04:00:24,600 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676088016466 2023-02-11 04:00:24,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] procedure2.ProcedureExecutor(1029): Stored pid=76, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:24,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-11 04:00:24,605 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088024605"}]},"ts":"1676088024605"} 2023-02-11 04:00:24,607 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676088016466, state=DISABLING in hbase:meta 2023-02-11 04:00:24,609 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676088016466 to state=DISABLING 2023-02-11 04:00:24,610 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=77, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, UNASSIGN}, {pid=78, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, UNASSIGN}, {pid=79, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, UNASSIGN}, {pid=80, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, UNASSIGN}, {pid=81, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, UNASSIGN}, {pid=82, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, UNASSIGN}] 2023-02-11 04:00:24,612 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=78, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, UNASSIGN 2023-02-11 04:00:24,612 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=77, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, UNASSIGN 2023-02-11 04:00:24,612 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=79, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, UNASSIGN 2023-02-11 04:00:24,612 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=80, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, UNASSIGN 2023-02-11 04:00:24,613 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=82, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, UNASSIGN 2023-02-11 04:00:24,613 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=78 updating hbase:meta row=3452669615c81575c52ff96406a41e9e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:24,613 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088024613"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088024613"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088024613"}]},"ts":"1676088024613"} 2023-02-11 04:00:24,613 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=e1d0ef015d92b032f67c5052a11865c4, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:24,613 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=79 updating hbase:meta row=51e4d6a6293430d21cdf75126c54f808, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:24,614 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088024613"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088024613"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088024613"}]},"ts":"1676088024613"} 2023-02-11 04:00:24,614 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088024613"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088024613"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088024613"}]},"ts":"1676088024613"} 2023-02-11 04:00:24,613 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=77 updating hbase:meta row=747a6fa324aa972cea6bbd307f3b1539, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:24,614 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088024613"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088024613"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088024613"}]},"ts":"1676088024613"} 2023-02-11 04:00:24,614 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=82 updating hbase:meta row=d586dffdbc612899aa5c72d99bdc6519, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:24,615 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088024614"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088024614"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088024614"}]},"ts":"1676088024614"} 2023-02-11 04:00:24,615 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=83, ppid=78, state=RUNNABLE; CloseRegionProcedure 3452669615c81575c52ff96406a41e9e, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:24,616 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=84, ppid=80, state=RUNNABLE; CloseRegionProcedure e1d0ef015d92b032f67c5052a11865c4, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:24,618 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=85, ppid=79, state=RUNNABLE; CloseRegionProcedure 51e4d6a6293430d21cdf75126c54f808, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:24,619 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=86, ppid=77, state=RUNNABLE; CloseRegionProcedure 747a6fa324aa972cea6bbd307f3b1539, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:24,620 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=87, ppid=82, state=RUNNABLE; CloseRegionProcedure d586dffdbc612899aa5c72d99bdc6519, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:24,621 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=81, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, UNASSIGN 2023-02-11 04:00:24,621 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=aaa51003a74d8671fa0840bbfcb549ac, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:24,622 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088024621"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088024621"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088024621"}]},"ts":"1676088024621"} 2023-02-11 04:00:24,623 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=88, ppid=81, state=RUNNABLE; CloseRegionProcedure aaa51003a74d8671fa0840bbfcb549ac, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:24,769 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:24,770 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 3452669615c81575c52ff96406a41e9e, disabling compactions & flushes 2023-02-11 04:00:24,770 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:24,770 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:24,770 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. after waiting 0 ms 2023-02-11 04:00:24,770 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:24,770 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 3452669615c81575c52ff96406a41e9e 1/1 column families, dataSize=2.16 KB heapSize=4.89 KB 2023-02-11 04:00:24,772 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:24,772 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:24,772 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 747a6fa324aa972cea6bbd307f3b1539, disabling compactions & flushes 2023-02-11 04:00:24,773 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 51e4d6a6293430d21cdf75126c54f808, disabling compactions & flushes 2023-02-11 04:00:24,773 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:24,773 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:24,773 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. after waiting 0 ms 2023-02-11 04:00:24,773 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:24,773 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 51e4d6a6293430d21cdf75126c54f808 1/1 column families, dataSize=2.22 KB heapSize=5.03 KB 2023-02-11 04:00:24,773 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:24,774 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:24,774 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. after waiting 0 ms 2023-02-11 04:00:24,774 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:24,774 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 747a6fa324aa972cea6bbd307f3b1539 1/1 column families, dataSize=2.16 KB heapSize=4.89 KB 2023-02-11 04:00:24,811 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 2023-02-11 04:00:24,811 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 2023-02-11 04:00:24,825 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b 2023-02-11 04:00:24,829 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/.tmp/cf/c1e521afb6ba4af19d964c416aa5a0de 2023-02-11 04:00:24,829 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/.tmp/cf/f60cf166ab3e4854a8bb6a91f3df546f 2023-02-11 04:00:24,833 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/.tmp/cf/c1e521afb6ba4af19d964c416aa5a0de as hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/c1e521afb6ba4af19d964c416aa5a0de 2023-02-11 04:00:24,833 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/.tmp/cf/f60cf166ab3e4854a8bb6a91f3df546f as hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/f60cf166ab3e4854a8bb6a91f3df546f 2023-02-11 04:00:24,837 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/f60cf166ab3e4854a8bb6a91f3df546f, entries=33, sequenceid=15, filesize=10.5 K 2023-02-11 04:00:24,837 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/.tmp/cf/7dc891a823984ca99b9b428443d59643 2023-02-11 04:00:24,837 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/c1e521afb6ba4af19d964c416aa5a0de, entries=34, sequenceid=15, filesize=10.7 K 2023-02-11 04:00:24,838 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.16 KB/2209, heapSize ~4.88 KB/4992, currentSize=0 B/0 for 3452669615c81575c52ff96406a41e9e in 68ms, sequenceid=15, compaction requested=false 2023-02-11 04:00:24,838 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.22 KB/2276, heapSize ~5.02 KB/5136, currentSize=0 B/0 for 51e4d6a6293430d21cdf75126c54f808 in 65ms, sequenceid=15, compaction requested=false 2023-02-11 04:00:24,847 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/.tmp/cf/7dc891a823984ca99b9b428443d59643 as hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/7dc891a823984ca99b9b428443d59643 2023-02-11 04:00:24,848 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 04:00:24,849 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 04:00:24,850 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:24,850 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:24,850 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 3452669615c81575c52ff96406a41e9e: Waiting for close lock at 1676088024769Running coprocessor pre-close hooks at 1676088024769Disabling compacts and flushes for region at 1676088024769Disabling writes for close at 1676088024770 (+1 ms)Obtaining lock to block concurrent updates at 1676088024770Preparing flush snapshotting stores in 3452669615c81575c52ff96406a41e9e at 1676088024770Finished memstore snapshotting testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e., syncing WAL and waiting on mvcc, flushsize=dataSize=2209, getHeapSize=4992, getOffHeapSize=0, getCellsCount=33 at 1676088024770Flushing stores of testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. at 1676088024771 (+1 ms)Flushing 3452669615c81575c52ff96406a41e9e/cf: creating writer at 1676088024772 (+1 ms)Flushing mob file 3452669615c81575c52ff96406a41e9e/cf: appending metadata at 1676088024782 (+10 ms)Flushing mob file 3452669615c81575c52ff96406a41e9e/cf: closing flushed file at 1676088024782Flushing 3452669615c81575c52ff96406a41e9e/cf: appending metadata at 1676088024812 (+30 ms)Flushing 3452669615c81575c52ff96406a41e9e/cf: closing flushed file at 1676088024812Flushing 3452669615c81575c52ff96406a41e9e/cf: reopening flushed file at 1676088024833 (+21 ms)Finished flush of dataSize ~2.16 KB/2209, heapSize ~4.88 KB/4992, currentSize=0 B/0 for 3452669615c81575c52ff96406a41e9e in 68ms, sequenceid=15, compaction requested=false at 1676088024838 (+5 ms)Writing region close event to WAL at 1676088024842 (+4 ms)Running coprocessor post-close hooks at 1676088024849 (+7 ms)Closed at 1676088024850 (+1 ms) 2023-02-11 04:00:24,850 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 51e4d6a6293430d21cdf75126c54f808: Waiting for close lock at 1676088024772Running coprocessor pre-close hooks at 1676088024772Disabling compacts and flushes for region at 1676088024772Disabling writes for close at 1676088024773 (+1 ms)Obtaining lock to block concurrent updates at 1676088024773Preparing flush snapshotting stores in 51e4d6a6293430d21cdf75126c54f808 at 1676088024773Finished memstore snapshotting testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808., syncing WAL and waiting on mvcc, flushsize=dataSize=2276, getHeapSize=5136, getOffHeapSize=0, getCellsCount=34 at 1676088024774 (+1 ms)Flushing stores of testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. at 1676088024775 (+1 ms)Flushing 51e4d6a6293430d21cdf75126c54f808/cf: creating writer at 1676088024775Flushing mob file 51e4d6a6293430d21cdf75126c54f808/cf: appending metadata at 1676088024782 (+7 ms)Flushing mob file 51e4d6a6293430d21cdf75126c54f808/cf: closing flushed file at 1676088024782Flushing 51e4d6a6293430d21cdf75126c54f808/cf: appending metadata at 1676088024812 (+30 ms)Flushing 51e4d6a6293430d21cdf75126c54f808/cf: closing flushed file at 1676088024812Flushing 51e4d6a6293430d21cdf75126c54f808/cf: reopening flushed file at 1676088024833 (+21 ms)Finished flush of dataSize ~2.22 KB/2276, heapSize ~5.02 KB/5136, currentSize=0 B/0 for 51e4d6a6293430d21cdf75126c54f808 in 65ms, sequenceid=15, compaction requested=false at 1676088024838 (+5 ms)Writing region close event to WAL at 1676088024843 (+5 ms)Running coprocessor post-close hooks at 1676088024850 (+7 ms)Closed at 1676088024850 2023-02-11 04:00:24,852 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:24,852 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:24,853 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing aaa51003a74d8671fa0840bbfcb549ac, disabling compactions & flushes 2023-02-11 04:00:24,853 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:24,853 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:24,853 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. after waiting 0 ms 2023-02-11 04:00:24,853 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/7dc891a823984ca99b9b428443d59643, entries=33, sequenceid=15, filesize=10.5 K 2023-02-11 04:00:24,853 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:24,853 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing aaa51003a74d8671fa0840bbfcb549ac 1/1 column families, dataSize=1.57 KB heapSize=3.63 KB 2023-02-11 04:00:24,854 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=79 updating hbase:meta row=51e4d6a6293430d21cdf75126c54f808, regionState=CLOSED 2023-02-11 04:00:24,854 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088024854"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088024854"}]},"ts":"1676088024854"} 2023-02-11 04:00:24,854 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:24,854 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:24,855 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e1d0ef015d92b032f67c5052a11865c4, disabling compactions & flushes 2023-02-11 04:00:24,855 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:24,855 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:24,855 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. after waiting 0 ms 2023-02-11 04:00:24,855 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:24,855 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing e1d0ef015d92b032f67c5052a11865c4 1/1 column families, dataSize=1.83 KB heapSize=4.19 KB 2023-02-11 04:00:24,855 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.16 KB/2209, heapSize ~4.88 KB/4992, currentSize=0 B/0 for 747a6fa324aa972cea6bbd307f3b1539 in 81ms, sequenceid=15, compaction requested=false 2023-02-11 04:00:24,855 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=78 updating hbase:meta row=3452669615c81575c52ff96406a41e9e, regionState=CLOSED 2023-02-11 04:00:24,855 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088024855"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088024855"}]},"ts":"1676088024855"} 2023-02-11 04:00:24,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-11 04:00:24,865 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 04:00:24,865 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=85, resume processing ppid=79 2023-02-11 04:00:24,865 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=85, ppid=79, state=SUCCESS; CloseRegionProcedure 51e4d6a6293430d21cdf75126c54f808, server=jenkins-hbase4.apache.org,46851,1676088013694 in 243 msec 2023-02-11 04:00:24,866 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=83, resume processing ppid=78 2023-02-11 04:00:24,866 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=83, ppid=78, state=SUCCESS; CloseRegionProcedure 3452669615c81575c52ff96406a41e9e, server=jenkins-hbase4.apache.org,40829,1676088013739 in 247 msec 2023-02-11 04:00:24,867 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=79, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, UNASSIGN in 255 msec 2023-02-11 04:00:24,867 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=78, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, UNASSIGN in 256 msec 2023-02-11 04:00:24,872 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:24,873 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 747a6fa324aa972cea6bbd307f3b1539: Waiting for close lock at 1676088024772Running coprocessor pre-close hooks at 1676088024772Disabling compacts and flushes for region at 1676088024772Disabling writes for close at 1676088024774 (+2 ms)Obtaining lock to block concurrent updates at 1676088024775 (+1 ms)Preparing flush snapshotting stores in 747a6fa324aa972cea6bbd307f3b1539 at 1676088024775Finished memstore snapshotting testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539., syncing WAL and waiting on mvcc, flushsize=dataSize=2209, getHeapSize=4992, getOffHeapSize=0, getCellsCount=33 at 1676088024775Flushing stores of testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. at 1676088024776 (+1 ms)Flushing 747a6fa324aa972cea6bbd307f3b1539/cf: creating writer at 1676088024776Flushing mob file 747a6fa324aa972cea6bbd307f3b1539/cf: appending metadata at 1676088024789 (+13 ms)Flushing mob file 747a6fa324aa972cea6bbd307f3b1539/cf: closing flushed file at 1676088024789Flushing 747a6fa324aa972cea6bbd307f3b1539/cf: appending metadata at 1676088024826 (+37 ms)Flushing 747a6fa324aa972cea6bbd307f3b1539/cf: closing flushed file at 1676088024826Flushing 747a6fa324aa972cea6bbd307f3b1539/cf: reopening flushed file at 1676088024848 (+22 ms)Finished flush of dataSize ~2.16 KB/2209, heapSize ~4.88 KB/4992, currentSize=0 B/0 for 747a6fa324aa972cea6bbd307f3b1539 in 81ms, sequenceid=15, compaction requested=false at 1676088024855 (+7 ms)Writing region close event to WAL at 1676088024860 (+5 ms)Running coprocessor post-close hooks at 1676088024865 (+5 ms)Closed at 1676088024872 (+7 ms) 2023-02-11 04:00:24,874 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:24,874 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:24,874 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing d586dffdbc612899aa5c72d99bdc6519, disabling compactions & flushes 2023-02-11 04:00:24,874 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:24,874 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:24,874 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. after waiting 0 ms 2023-02-11 04:00:24,875 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:24,875 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing d586dffdbc612899aa5c72d99bdc6519 1/1 column families, dataSize=22.77 KB heapSize=49.19 KB 2023-02-11 04:00:24,875 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=77 updating hbase:meta row=747a6fa324aa972cea6bbd307f3b1539, regionState=CLOSED 2023-02-11 04:00:24,875 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088024875"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088024875"}]},"ts":"1676088024875"} 2023-02-11 04:00:24,879 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=86, resume processing ppid=77 2023-02-11 04:00:24,879 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=86, ppid=77, state=SUCCESS; CloseRegionProcedure 747a6fa324aa972cea6bbd307f3b1539, server=jenkins-hbase4.apache.org,42385,1676088013771 in 258 msec 2023-02-11 04:00:24,880 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=77, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, UNASSIGN in 269 msec 2023-02-11 04:00:24,895 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 2023-02-11 04:00:24,898 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f 2023-02-11 04:00:24,914 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/.tmp/cf/e6cd8a1c78d94d0c9da6aff3413bb91a 2023-02-11 04:00:24,916 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/.tmp/cf/e1f68e07f6354647bc060c949ab8bd60 2023-02-11 04:00:24,916 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e 2023-02-11 04:00:24,918 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/.tmp/cf/e6cd8a1c78d94d0c9da6aff3413bb91a as hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/e6cd8a1c78d94d0c9da6aff3413bb91a 2023-02-11 04:00:24,920 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/.tmp/cf/e1f68e07f6354647bc060c949ab8bd60 as hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/e1f68e07f6354647bc060c949ab8bd60 2023-02-11 04:00:24,923 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/e6cd8a1c78d94d0c9da6aff3413bb91a, entries=24, sequenceid=15, filesize=9.0 K 2023-02-11 04:00:24,924 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.57 KB/1606, heapSize ~3.61 KB/3696, currentSize=0 B/0 for aaa51003a74d8671fa0840bbfcb549ac in 71ms, sequenceid=15, compaction requested=false 2023-02-11 04:00:24,924 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/e1f68e07f6354647bc060c949ab8bd60, entries=28, sequenceid=15, filesize=9.7 K 2023-02-11 04:00:24,928 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.83 KB/1874, heapSize ~4.17 KB/4272, currentSize=0 B/0 for e1d0ef015d92b032f67c5052a11865c4 in 73ms, sequenceid=15, compaction requested=false 2023-02-11 04:00:24,933 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=22.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/.tmp/cf/d1e1bf338c724083a10375a4bd78b7d9 2023-02-11 04:00:24,934 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 04:00:24,935 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:24,935 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for aaa51003a74d8671fa0840bbfcb549ac: Waiting for close lock at 1676088024852Running coprocessor pre-close hooks at 1676088024852Disabling compacts and flushes for region at 1676088024852Disabling writes for close at 1676088024853 (+1 ms)Obtaining lock to block concurrent updates at 1676088024853Preparing flush snapshotting stores in aaa51003a74d8671fa0840bbfcb549ac at 1676088024853Finished memstore snapshotting testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac., syncing WAL and waiting on mvcc, flushsize=dataSize=1606, getHeapSize=3696, getOffHeapSize=0, getCellsCount=24 at 1676088024854 (+1 ms)Flushing stores of testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. at 1676088024855 (+1 ms)Flushing aaa51003a74d8671fa0840bbfcb549ac/cf: creating writer at 1676088024856 (+1 ms)Flushing mob file aaa51003a74d8671fa0840bbfcb549ac/cf: appending metadata at 1676088024866 (+10 ms)Flushing mob file aaa51003a74d8671fa0840bbfcb549ac/cf: closing flushed file at 1676088024866Flushing aaa51003a74d8671fa0840bbfcb549ac/cf: appending metadata at 1676088024899 (+33 ms)Flushing aaa51003a74d8671fa0840bbfcb549ac/cf: closing flushed file at 1676088024899Flushing aaa51003a74d8671fa0840bbfcb549ac/cf: reopening flushed file at 1676088024919 (+20 ms)Finished flush of dataSize ~1.57 KB/1606, heapSize ~3.61 KB/3696, currentSize=0 B/0 for aaa51003a74d8671fa0840bbfcb549ac in 71ms, sequenceid=15, compaction requested=false at 1676088024924 (+5 ms)Writing region close event to WAL at 1676088024928 (+4 ms)Running coprocessor post-close hooks at 1676088024934 (+6 ms)Closed at 1676088024935 (+1 ms) 2023-02-11 04:00:24,937 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:24,937 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 04:00:24,938 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=aaa51003a74d8671fa0840bbfcb549ac, regionState=CLOSED 2023-02-11 04:00:24,938 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088024938"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088024938"}]},"ts":"1676088024938"} 2023-02-11 04:00:24,938 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:24,939 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e1d0ef015d92b032f67c5052a11865c4: Waiting for close lock at 1676088024854Running coprocessor pre-close hooks at 1676088024854Disabling compacts and flushes for region at 1676088024854Disabling writes for close at 1676088024855 (+1 ms)Obtaining lock to block concurrent updates at 1676088024855Preparing flush snapshotting stores in e1d0ef015d92b032f67c5052a11865c4 at 1676088024855Finished memstore snapshotting testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4., syncing WAL and waiting on mvcc, flushsize=dataSize=1874, getHeapSize=4272, getOffHeapSize=0, getCellsCount=28 at 1676088024855Flushing stores of testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. at 1676088024859 (+4 ms)Flushing e1d0ef015d92b032f67c5052a11865c4/cf: creating writer at 1676088024860 (+1 ms)Flushing mob file e1d0ef015d92b032f67c5052a11865c4/cf: appending metadata at 1676088024865 (+5 ms)Flushing mob file e1d0ef015d92b032f67c5052a11865c4/cf: closing flushed file at 1676088024865Flushing e1d0ef015d92b032f67c5052a11865c4/cf: appending metadata at 1676088024898 (+33 ms)Flushing e1d0ef015d92b032f67c5052a11865c4/cf: closing flushed file at 1676088024898Flushing e1d0ef015d92b032f67c5052a11865c4/cf: reopening flushed file at 1676088024921 (+23 ms)Finished flush of dataSize ~1.83 KB/1874, heapSize ~4.17 KB/4272, currentSize=0 B/0 for e1d0ef015d92b032f67c5052a11865c4 in 73ms, sequenceid=15, compaction requested=false at 1676088024928 (+7 ms)Writing region close event to WAL at 1676088024933 (+5 ms)Running coprocessor post-close hooks at 1676088024938 (+5 ms)Closed at 1676088024938 2023-02-11 04:00:24,939 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/.tmp/cf/d1e1bf338c724083a10375a4bd78b7d9 as hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/d1e1bf338c724083a10375a4bd78b7d9 2023-02-11 04:00:24,940 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:24,942 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=e1d0ef015d92b032f67c5052a11865c4, regionState=CLOSED 2023-02-11 04:00:24,942 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088024941"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088024941"}]},"ts":"1676088024941"} 2023-02-11 04:00:24,945 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/d1e1bf338c724083a10375a4bd78b7d9, entries=348, sequenceid=15, filesize=64.2 K 2023-02-11 04:00:24,945 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~22.77 KB/23312, heapSize ~49.17 KB/50352, currentSize=0 B/0 for d586dffdbc612899aa5c72d99bdc6519 in 70ms, sequenceid=15, compaction requested=false 2023-02-11 04:00:24,951 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=88, resume processing ppid=81 2023-02-11 04:00:24,951 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=88, ppid=81, state=SUCCESS; CloseRegionProcedure aaa51003a74d8671fa0840bbfcb549ac, server=jenkins-hbase4.apache.org,46851,1676088013694 in 318 msec 2023-02-11 04:00:24,953 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=84, resume processing ppid=80 2023-02-11 04:00:24,953 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=84, ppid=80, state=SUCCESS; CloseRegionProcedure e1d0ef015d92b032f67c5052a11865c4, server=jenkins-hbase4.apache.org,40829,1676088013739 in 328 msec 2023-02-11 04:00:24,953 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=81, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, UNASSIGN in 341 msec 2023-02-11 04:00:24,955 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=80, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, UNASSIGN in 343 msec 2023-02-11 04:00:24,955 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 04:00:24,956 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:24,957 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for d586dffdbc612899aa5c72d99bdc6519: Waiting for close lock at 1676088024874Running coprocessor pre-close hooks at 1676088024874Disabling compacts and flushes for region at 1676088024874Disabling writes for close at 1676088024874Obtaining lock to block concurrent updates at 1676088024875 (+1 ms)Preparing flush snapshotting stores in d586dffdbc612899aa5c72d99bdc6519 at 1676088024875Finished memstore snapshotting testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519., syncing WAL and waiting on mvcc, flushsize=dataSize=23312, getHeapSize=50352, getOffHeapSize=0, getCellsCount=348 at 1676088024875Flushing stores of testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. at 1676088024875Flushing d586dffdbc612899aa5c72d99bdc6519/cf: creating writer at 1676088024887 (+12 ms)Flushing mob file d586dffdbc612899aa5c72d99bdc6519/cf: appending metadata at 1676088024896 (+9 ms)Flushing mob file d586dffdbc612899aa5c72d99bdc6519/cf: closing flushed file at 1676088024896Flushing d586dffdbc612899aa5c72d99bdc6519/cf: appending metadata at 1676088024917 (+21 ms)Flushing d586dffdbc612899aa5c72d99bdc6519/cf: closing flushed file at 1676088024917Flushing d586dffdbc612899aa5c72d99bdc6519/cf: reopening flushed file at 1676088024940 (+23 ms)Finished flush of dataSize ~22.77 KB/23312, heapSize ~49.17 KB/50352, currentSize=0 B/0 for d586dffdbc612899aa5c72d99bdc6519 in 70ms, sequenceid=15, compaction requested=false at 1676088024946 (+6 ms)Writing region close event to WAL at 1676088024951 (+5 ms)Running coprocessor post-close hooks at 1676088024956 (+5 ms)Closed at 1676088024956 2023-02-11 04:00:24,958 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:24,959 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=82 updating hbase:meta row=d586dffdbc612899aa5c72d99bdc6519, regionState=CLOSED 2023-02-11 04:00:24,959 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088024959"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088024959"}]},"ts":"1676088024959"} 2023-02-11 04:00:24,963 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=87, resume processing ppid=82 2023-02-11 04:00:24,963 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=87, ppid=82, state=SUCCESS; CloseRegionProcedure d586dffdbc612899aa5c72d99bdc6519, server=jenkins-hbase4.apache.org,42385,1676088013771 in 341 msec 2023-02-11 04:00:24,965 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=82, resume processing ppid=76 2023-02-11 04:00:24,965 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=82, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, UNASSIGN in 353 msec 2023-02-11 04:00:24,966 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088024966"}]},"ts":"1676088024966"} 2023-02-11 04:00:24,967 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676088016466, state=DISABLED in hbase:meta 2023-02-11 04:00:24,969 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676088016466 to state=DISABLED 2023-02-11 04:00:24,971 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=76, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676088016466 in 370 msec 2023-02-11 04:00:25,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-11 04:00:25,360 INFO [Listener at localhost/33753] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676088016466, procId: 76 completed 2023-02-11 04:00:25,361 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=snaptb1-1676088016466 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 } 2023-02-11 04:00:25,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676088025361 (current time:1676088025361). 2023-02-11 04:00:25,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-11 04:00:25,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] zookeeper.ReadOnlyZKClient(139): Connect 0x490d1f64 to 127.0.0.1:63425 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 04:00:25,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69a20446, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 04:00:25,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 04:00:25,371 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47722, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 04:00:25,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x490d1f64 to 127.0.0.1:63425 2023-02-11 04:00:25,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:00:25,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-11 04:00:25,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb1-1676088016466 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 } 2023-02-11 04:00:25,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] procedure2.ProcedureExecutor(1029): Stored pid=89, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676088016466, type=EXCLUSIVE 2023-02-11 04:00:25,376 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=89, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676088016466, type=EXCLUSIVE 2023-02-11 04:00:25,376 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=89, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676088016466, type=EXCLUSIVE; timeout=600000, timestamp=1676088625376 2023-02-11 04:00:25,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb1-1676088016466 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 } 2023-02-11 04:00:25,376 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb1-1676088016466 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676088016466 2023-02-11 04:00:25,377 DEBUG [Listener at localhost/33753] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb1-1676088016466 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-11 04:00:25,377 DEBUG [Listener at localhost/33753] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-11 04:00:25,394 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb1-1676088016466 table=testRestoreSnapshot-1676088016466 type=DISABLED ttl=0 } 2023-02-11 04:00:25,398 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:25,399 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:25,400 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:25,400 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:25,400 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:25,400 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:25,400 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:25,400 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:25,401 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:25,401 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:25,401 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:25,401 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:25,402 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:25,401 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:25,403 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/93d3b4e383ae4310a66075c83ee5c664, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/e1f68e07f6354647bc060c949ab8bd60] hfiles 2023-02-11 04:00:25,403 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/93d3b4e383ae4310a66075c83ee5c664 2023-02-11 04:00:25,403 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/e1f68e07f6354647bc060c949ab8bd60 2023-02-11 04:00:25,405 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/a11c828e49bd4e7a9cf05be9ff07ad7f, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/f60cf166ab3e4854a8bb6a91f3df546f] hfiles 2023-02-11 04:00:25,405 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/a11c828e49bd4e7a9cf05be9ff07ad7f 2023-02-11 04:00:25,405 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/f60cf166ab3e4854a8bb6a91f3df546f 2023-02-11 04:00:25,406 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/3f27b1f3105b45bf81c17ba9c704bf44, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/e6cd8a1c78d94d0c9da6aff3413bb91a] hfiles 2023-02-11 04:00:25,406 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/3f27b1f3105b45bf81c17ba9c704bf44 2023-02-11 04:00:25,406 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/e6cd8a1c78d94d0c9da6aff3413bb91a 2023-02-11 04:00:25,407 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/0a646c34d29c4963afc688a0da1fd6d4, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/7dc891a823984ca99b9b428443d59643] hfiles 2023-02-11 04:00:25,407 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/0a646c34d29c4963afc688a0da1fd6d4 2023-02-11 04:00:25,407 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/7dc891a823984ca99b9b428443d59643 2023-02-11 04:00:25,409 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/d1e1bf338c724083a10375a4bd78b7d9, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/e928d1188fee412e9c504ba31bbb3ef6] hfiles 2023-02-11 04:00:25,409 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/d1e1bf338c724083a10375a4bd78b7d9 2023-02-11 04:00:25,409 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/e928d1188fee412e9c504ba31bbb3ef6 2023-02-11 04:00:25,412 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/1b38064ae3304eca9c176d5669359823, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/c1e521afb6ba4af19d964c416aa5a0de] hfiles 2023-02-11 04:00:25,412 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/1b38064ae3304eca9c176d5669359823 2023-02-11 04:00:25,412 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/c1e521afb6ba4af19d964c416aa5a0de 2023-02-11 04:00:25,415 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45] hfiles 2023-02-11 04:00:25,416 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/12): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f 2023-02-11 04:00:25,416 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/12): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 2023-02-11 04:00:25,416 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (3/12): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 2023-02-11 04:00:25,416 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (4/12): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 2023-02-11 04:00:25,416 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (5/12): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 2023-02-11 04:00:25,416 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (6/12): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 2023-02-11 04:00:25,416 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (7/12): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 2023-02-11 04:00:25,416 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (8/12): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b 2023-02-11 04:00:25,416 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (9/12): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 2023-02-11 04:00:25,416 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (10/12): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e 2023-02-11 04:00:25,416 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (11/12): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 2023-02-11 04:00:25,416 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (12/12): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 2023-02-11 04:00:25,493 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x490facbf to 127.0.0.1:63425 2023-02-11 04:00:25,493 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:00:25,509 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb1-1676088016466 table=testRestoreSnapshot-1676088016466 type=DISABLED ttl=0 } as finished. 2023-02-11 04:00:25,509 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb1-1676088016466 2023-02-11 04:00:25,510 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/.tmp/snaptb1-1676088016466 2023-02-11 04:00:25,548 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/.tmp/snaptb1-1676088016466 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/snaptb1-1676088016466 2023-02-11 04:00:25,571 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb1-1676088016466 of table testRestoreSnapshot-1676088016466 completed 2023-02-11 04:00:25,571 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/.tmp/snaptb1-1676088016466 2023-02-11 04:00:25,571 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/.tmp/snaptb1-1676088016466 2023-02-11 04:00:25,571 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb1-1676088016466 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676088016466 at 1676088025377Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb1-1676088016466 table=testRestoreSnapshot-1676088016466 type=DISABLED ttl=0 } at 1676088025394 (+17 ms)Consolidate snapshot: snaptb1-1676088016466 at 1676088025509 (+115 ms)Loading Region manifests for snaptb1-1676088016466 at 1676088025509Writing data manifest for snaptb1-1676088016466 at 1676088025515 (+6 ms)Verifying snapshot: snaptb1-1676088016466 at 1676088025533 (+18 ms)Snapshot snaptb1-1676088016466 of table testRestoreSnapshot-1676088016466 completed at 1676088025571 (+38 ms) 2023-02-11 04:00:25,573 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=89, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676088016466, type=EXCLUSIVE 2023-02-11 04:00:25,574 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=89, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676088016466, type=EXCLUSIVE in 199 msec 2023-02-11 04:00:25,627 DEBUG [Listener at localhost/33753] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 04:00:25,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb1-1676088016466 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 } is done 2023-02-11 04:00:25,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb1-1676088016466 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-11 04:00:25,647 INFO [Listener at localhost/33753] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-snaptb0-1676088016466-1676088025647 2023-02-11 04:00:25,648 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=hbase-failsafe-snaptb0-1676088016466-1676088025647 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 } 2023-02-11 04:00:25,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676088025648 (current time:1676088025648). 2023-02-11 04:00:25,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-11 04:00:25,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] zookeeper.ReadOnlyZKClient(139): Connect 0x2eec2b0c to 127.0.0.1:63425 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 04:00:25,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@77e98b8b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 04:00:25,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 04:00:25,657 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47738, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 04:00:25,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2eec2b0c to 127.0.0.1:63425 2023-02-11 04:00:25,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:00:25,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-11 04:00:25,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-snaptb0-1676088016466-1676088025647 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 } 2023-02-11 04:00:25,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] procedure2.ProcedureExecutor(1029): Stored pid=90, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676088016466, type=EXCLUSIVE 2023-02-11 04:00:25,662 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=90, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676088016466, type=EXCLUSIVE 2023-02-11 04:00:25,663 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=90, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676088016466, type=EXCLUSIVE; timeout=600000, timestamp=1676088625663 2023-02-11 04:00:25,663 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-snaptb0-1676088016466-1676088025647 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676088016466 2023-02-11 04:00:25,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-snaptb0-1676088016466-1676088025647 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 } 2023-02-11 04:00:25,664 DEBUG [Listener at localhost/33753] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-snaptb0-1676088016466-1676088025647 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-11 04:00:25,664 DEBUG [Listener at localhost/33753] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-11 04:00:25,676 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb0-1676088016466-1676088025647 table=testRestoreSnapshot-1676088016466 type=DISABLED ttl=0 } 2023-02-11 04:00:25,678 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:25,678 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:25,678 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:25,678 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:25,679 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:25,679 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:25,680 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:25,680 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:25,680 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:25,680 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:25,680 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:25,680 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:25,681 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:25,681 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:25,685 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/1b38064ae3304eca9c176d5669359823, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/c1e521afb6ba4af19d964c416aa5a0de] hfiles 2023-02-11 04:00:25,686 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/1b38064ae3304eca9c176d5669359823 2023-02-11 04:00:25,686 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/c1e521afb6ba4af19d964c416aa5a0de 2023-02-11 04:00:25,686 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/93d3b4e383ae4310a66075c83ee5c664, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/e1f68e07f6354647bc060c949ab8bd60] hfiles 2023-02-11 04:00:25,686 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/93d3b4e383ae4310a66075c83ee5c664 2023-02-11 04:00:25,686 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/e1f68e07f6354647bc060c949ab8bd60 2023-02-11 04:00:25,686 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/a11c828e49bd4e7a9cf05be9ff07ad7f, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/f60cf166ab3e4854a8bb6a91f3df546f] hfiles 2023-02-11 04:00:25,686 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/a11c828e49bd4e7a9cf05be9ff07ad7f 2023-02-11 04:00:25,687 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/d1e1bf338c724083a10375a4bd78b7d9, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/e928d1188fee412e9c504ba31bbb3ef6] hfiles 2023-02-11 04:00:25,687 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/d1e1bf338c724083a10375a4bd78b7d9 2023-02-11 04:00:25,687 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/e928d1188fee412e9c504ba31bbb3ef6 2023-02-11 04:00:25,687 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/f60cf166ab3e4854a8bb6a91f3df546f 2023-02-11 04:00:25,688 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/3f27b1f3105b45bf81c17ba9c704bf44, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/e6cd8a1c78d94d0c9da6aff3413bb91a] hfiles 2023-02-11 04:00:25,688 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/3f27b1f3105b45bf81c17ba9c704bf44 2023-02-11 04:00:25,688 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/e6cd8a1c78d94d0c9da6aff3413bb91a 2023-02-11 04:00:25,691 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/0a646c34d29c4963afc688a0da1fd6d4, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/7dc891a823984ca99b9b428443d59643] hfiles 2023-02-11 04:00:25,691 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/0a646c34d29c4963afc688a0da1fd6d4 2023-02-11 04:00:25,691 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/7dc891a823984ca99b9b428443d59643 2023-02-11 04:00:25,696 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45] hfiles 2023-02-11 04:00:25,697 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/12): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f 2023-02-11 04:00:25,697 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/12): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 2023-02-11 04:00:25,697 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (3/12): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 2023-02-11 04:00:25,697 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (4/12): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 2023-02-11 04:00:25,697 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (5/12): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 2023-02-11 04:00:25,697 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (6/12): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 2023-02-11 04:00:25,697 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (7/12): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 2023-02-11 04:00:25,697 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (8/12): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b 2023-02-11 04:00:25,697 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (9/12): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 2023-02-11 04:00:25,697 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (10/12): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e 2023-02-11 04:00:25,697 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (11/12): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 2023-02-11 04:00:25,697 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (12/12): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 2023-02-11 04:00:25,754 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-snaptb0-1676088016466-1676088025647 table=testRestoreSnapshot-1676088016466 type=DISABLED ttl=0 } as finished. 2023-02-11 04:00:25,754 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-snaptb0-1676088016466-1676088025647 2023-02-11 04:00:25,755 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676088016466-1676088025647 2023-02-11 04:00:25,785 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676088016466-1676088025647 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/hbase-failsafe-snaptb0-1676088016466-1676088025647 2023-02-11 04:00:25,806 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-snaptb0-1676088016466-1676088025647 of table testRestoreSnapshot-1676088016466 completed 2023-02-11 04:00:25,807 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676088016466-1676088025647 2023-02-11 04:00:25,807 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676088016466-1676088025647 2023-02-11 04:00:25,807 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-snaptb0-1676088016466-1676088025647 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676088016466 at 1676088025663Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb0-1676088016466-1676088025647 table=testRestoreSnapshot-1676088016466 type=DISABLED ttl=0 } at 1676088025676 (+13 ms)Consolidate snapshot: hbase-failsafe-snaptb0-1676088016466-1676088025647 at 1676088025754 (+78 ms)Loading Region manifests for hbase-failsafe-snaptb0-1676088016466-1676088025647 at 1676088025755 (+1 ms)Writing data manifest for hbase-failsafe-snaptb0-1676088016466-1676088025647 at 1676088025760 (+5 ms)Verifying snapshot: hbase-failsafe-snaptb0-1676088016466-1676088025647 at 1676088025772 (+12 ms)Snapshot hbase-failsafe-snaptb0-1676088016466-1676088025647 of table testRestoreSnapshot-1676088016466 completed at 1676088025806 (+34 ms) 2023-02-11 04:00:25,808 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=90, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676088016466, type=EXCLUSIVE 2023-02-11 04:00:25,810 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=90, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676088016466, type=EXCLUSIVE in 147 msec 2023-02-11 04:00:25,914 DEBUG [Listener at localhost/33753] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 04:00:25,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb0-1676088016466-1676088025647 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 } is done 2023-02-11 04:00:25,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-snaptb0-1676088016466-1676088025647 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-11 04:00:25,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] procedure2.ProcedureExecutor(1029): Stored pid=91, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676088016466 snapshot=name: "snaptb0-1676088016466" table: "testRestoreSnapshot-1676088016466" creation_time: 1676088019525 type: DISABLED version: 2 ttl: 0 ) 2023-02-11 04:00:25,948 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(880): Restore snapshot=snaptb0-1676088016466 as table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:25,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=91 2023-02-11 04:00:25,966 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/.tabledesc/.tableinfo.0000000002 2023-02-11 04:00:25,967 DEBUG [PEWorker-1] util.FSTableDescriptors(519): Deleted hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/.tabledesc/.tableinfo.0000000001 2023-02-11 04:00:25,967 INFO [PEWorker-1] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/.tabledesc/.tableinfo.0000000002 2023-02-11 04:00:25,968 INFO [PEWorker-1] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=snaptb0-1676088016466 table=testRestoreSnapshot-1676088016466 type=DISABLED ttl=0 } 2023-02-11 04:00:25,973 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb0-1676088016466" table: "testRestoreSnapshot-1676088016466" creation_time: 1676088019525 type: DISABLED version: 2 ttl: 0 2023-02-11 04:00:25,974 DEBUG [PEWorker-1] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466 2023-02-11 04:00:25,982 DEBUG [PEWorker-1] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:25,989 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f 2023-02-11 04:00:25,991 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 2023-02-11 04:00:25,992 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 2023-02-11 04:00:25,994 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e 2023-02-11 04:00:25,995 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 2023-02-11 04:00:25,996 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b 2023-02-11 04:00:25,996 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:25,997 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:25,997 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:25,997 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:25,997 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:25,997 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:26,004 DEBUG [RestoreSnapshot-pool-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/c1e521afb6ba4af19d964c416aa5a0de to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/c1e521afb6ba4af19d964c416aa5a0de 2023-02-11 04:00:26,005 DEBUG [RestoreSnapshot-pool-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/d1e1bf338c724083a10375a4bd78b7d9 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/d1e1bf338c724083a10375a4bd78b7d9 2023-02-11 04:00:26,006 DEBUG [RestoreSnapshot-pool-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/e1f68e07f6354647bc060c949ab8bd60 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/e1f68e07f6354647bc060c949ab8bd60 2023-02-11 04:00:26,006 DEBUG [RestoreSnapshot-pool-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/7dc891a823984ca99b9b428443d59643 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/7dc891a823984ca99b9b428443d59643 2023-02-11 04:00:26,007 DEBUG [RestoreSnapshot-pool-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/e6cd8a1c78d94d0c9da6aff3413bb91a to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/e6cd8a1c78d94d0c9da6aff3413bb91a 2023-02-11 04:00:26,026 DEBUG [RestoreSnapshot-pool-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/f60cf166ab3e4854a8bb6a91f3df546f to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/f60cf166ab3e4854a8bb6a91f3df546f 2023-02-11 04:00:26,026 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb0-1676088016466" table: "testRestoreSnapshot-1676088016466" creation_time: 1676088019525 type: DISABLED version: 2 ttl: 0 2023-02-11 04:00:26,029 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088026028"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:26,029 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088026028"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:26,029 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088026028"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:26,029 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088026028"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:26,029 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088026028"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:26,029 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088026028"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:26,036 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-11 04:00:26,036 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 3452669615c81575c52ff96406a41e9e, NAME => 'testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 51e4d6a6293430d21cdf75126c54f808, NAME => 'testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 747a6fa324aa972cea6bbd307f3b1539, NAME => 'testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => aaa51003a74d8671fa0840bbfcb549ac, NAME => 'testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => d586dffdbc612899aa5c72d99bdc6519, NAME => 'testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => e1d0ef015d92b032f67c5052a11865c4, NAME => 'testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.', STARTKEY => '3', ENDKEY => '4'}] 2023-02-11 04:00:26,037 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088026029"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088026029"}]},"ts":"1676088026029"} 2023-02-11 04:00:26,037 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088026029"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088026029"}]},"ts":"1676088026029"} 2023-02-11 04:00:26,037 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088026029"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088026029"}]},"ts":"1676088026029"} 2023-02-11 04:00:26,037 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088026029"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088026029"}]},"ts":"1676088026029"} 2023-02-11 04:00:26,037 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088026029"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088026029"}]},"ts":"1676088026029"} 2023-02-11 04:00:26,037 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088026029"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088026029"}]},"ts":"1676088026029"} 2023-02-11 04:00:26,040 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-11 04:00:26,040 INFO [PEWorker-1] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-11 04:00:26,040 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 3452669615c81575c52ff96406a41e9e, NAME => 'testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 51e4d6a6293430d21cdf75126c54f808, NAME => 'testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 747a6fa324aa972cea6bbd307f3b1539, NAME => 'testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => aaa51003a74d8671fa0840bbfcb549ac, NAME => 'testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => d586dffdbc612899aa5c72d99bdc6519, NAME => 'testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => e1d0ef015d92b032f67c5052a11865c4, NAME => 'testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.', STARTKEY => '3', ENDKEY => '4'}] 2023-02-11 04:00:26,041 INFO [PEWorker-1] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=snaptb0-1676088016466 table=testRestoreSnapshot-1676088016466 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676088016466 completed! 2023-02-11 04:00:26,044 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=91, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676088016466 snapshot=name: "snaptb0-1676088016466" table: "testRestoreSnapshot-1676088016466" creation_time: 1676088019525 type: DISABLED version: 2 ttl: 0 ) in 103 msec 2023-02-11 04:00:26,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=91 2023-02-11 04:00:26,203 INFO [Listener at localhost/33753] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676088016466, procId: 91 completed 2023-02-11 04:00:26,203 INFO [Listener at localhost/33753] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-snaptb0-1676088016466-1676088025647 2023-02-11 04:00:26,208 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "hbase-failsafe-snaptb0-1676088016466-1676088025647" 2023-02-11 04:00:26,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-snaptb0-1676088016466-1676088025647 2023-02-11 04:00:26,212 INFO [Listener at localhost/33753] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676088016466 2023-02-11 04:00:26,213 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676088016466 2023-02-11 04:00:26,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] procedure2.ProcedureExecutor(1029): Stored pid=92, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:26,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-11 04:00:26,217 INFO [PEWorker-3] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676088016466 2023-02-11 04:00:26,217 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088026217"}]},"ts":"1676088026217"} 2023-02-11 04:00:26,218 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676088016466, state=ENABLING in hbase:meta 2023-02-11 04:00:26,222 DEBUG [PEWorker-3] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-11 04:00:26,224 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, ASSIGN}, {pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, ASSIGN}, {pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, ASSIGN}, {pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, ASSIGN}, {pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, ASSIGN}, {pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, ASSIGN}] 2023-02-11 04:00:26,226 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, ASSIGN 2023-02-11 04:00:26,226 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, ASSIGN 2023-02-11 04:00:26,226 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, ASSIGN 2023-02-11 04:00:26,226 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, ASSIGN 2023-02-11 04:00:26,227 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, ASSIGN 2023-02-11 04:00:26,227 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 04:00:26,227 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 04:00:26,227 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 04:00:26,227 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 04:00:26,228 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 04:00:26,228 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, ASSIGN 2023-02-11 04:00:26,229 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 04:00:26,377 DEBUG [jenkins-hbase4:41713] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 04:00:26,378 DEBUG [jenkins-hbase4:41713] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 04:00:26,378 DEBUG [jenkins-hbase4:41713] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 04:00:26,378 DEBUG [jenkins-hbase4:41713] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 04:00:26,378 DEBUG [jenkins-hbase4:41713] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 04:00:26,381 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=94 updating hbase:meta row=3452669615c81575c52ff96406a41e9e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:26,381 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=747a6fa324aa972cea6bbd307f3b1539, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:26,381 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088026381"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088026381"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088026381"}]},"ts":"1676088026381"} 2023-02-11 04:00:26,381 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=d586dffdbc612899aa5c72d99bdc6519, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:26,381 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088026381"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088026381"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088026381"}]},"ts":"1676088026381"} 2023-02-11 04:00:26,381 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088026381"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088026381"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088026381"}]},"ts":"1676088026381"} 2023-02-11 04:00:26,381 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=96 updating hbase:meta row=e1d0ef015d92b032f67c5052a11865c4, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:26,382 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088026381"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088026381"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088026381"}]},"ts":"1676088026381"} 2023-02-11 04:00:26,381 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=aaa51003a74d8671fa0840bbfcb549ac, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:26,382 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088026381"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088026381"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088026381"}]},"ts":"1676088026381"} 2023-02-11 04:00:26,383 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=99, ppid=94, state=RUNNABLE; OpenRegionProcedure 3452669615c81575c52ff96406a41e9e, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:26,384 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=100, ppid=98, state=RUNNABLE; OpenRegionProcedure d586dffdbc612899aa5c72d99bdc6519, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:26,384 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=101, ppid=93, state=RUNNABLE; OpenRegionProcedure 747a6fa324aa972cea6bbd307f3b1539, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:26,385 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=102, ppid=96, state=RUNNABLE; OpenRegionProcedure e1d0ef015d92b032f67c5052a11865c4, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:26,386 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=103, ppid=97, state=RUNNABLE; OpenRegionProcedure aaa51003a74d8671fa0840bbfcb549ac, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:26,387 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=95 updating hbase:meta row=51e4d6a6293430d21cdf75126c54f808, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:26,387 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088026387"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088026387"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088026387"}]},"ts":"1676088026387"} 2023-02-11 04:00:26,389 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=104, ppid=95, state=RUNNABLE; OpenRegionProcedure 51e4d6a6293430d21cdf75126c54f808, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:26,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-11 04:00:26,538 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:26,539 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => aaa51003a74d8671fa0840bbfcb549ac, NAME => 'testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 04:00:26,539 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:26,539 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:26,539 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e1d0ef015d92b032f67c5052a11865c4, NAME => 'testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 04:00:26,539 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 51e4d6a6293430d21cdf75126c54f808, NAME => 'testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 04:00:26,539 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:26,539 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:26,539 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:26,539 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:26,539 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:26,539 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:26,540 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:26,539 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:26,540 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:26,540 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:26,540 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:26,540 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:26,541 INFO [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:26,541 INFO [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:26,541 INFO [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:26,542 DEBUG [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf 2023-02-11 04:00:26,542 DEBUG [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf 2023-02-11 04:00:26,542 DEBUG [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf 2023-02-11 04:00:26,542 DEBUG [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf 2023-02-11 04:00:26,542 DEBUG [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf 2023-02-11 04:00:26,542 DEBUG [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf 2023-02-11 04:00:26,542 INFO [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e1d0ef015d92b032f67c5052a11865c4 columnFamilyName cf 2023-02-11 04:00:26,543 INFO [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 51e4d6a6293430d21cdf75126c54f808 columnFamilyName cf 2023-02-11 04:00:26,543 INFO [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region aaa51003a74d8671fa0840bbfcb549ac columnFamilyName cf 2023-02-11 04:00:26,547 DEBUG [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/3f27b1f3105b45bf81c17ba9c704bf44 2023-02-11 04:00:26,548 DEBUG [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/93d3b4e383ae4310a66075c83ee5c664 2023-02-11 04:00:26,548 INFO [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] regionserver.HStore(310): Store=aaa51003a74d8671fa0840bbfcb549ac/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:26,548 DEBUG [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/1b38064ae3304eca9c176d5669359823 2023-02-11 04:00:26,548 INFO [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] regionserver.HStore(310): Store=e1d0ef015d92b032f67c5052a11865c4/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:26,548 INFO [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] regionserver.HStore(310): Store=51e4d6a6293430d21cdf75126c54f808/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:26,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:26,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:26,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:26,550 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:26,550 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:26,550 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:26,553 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:26,553 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:26,554 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:26,554 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 51e4d6a6293430d21cdf75126c54f808; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74877822, jitterRate=0.1157664954662323}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:26,554 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened aaa51003a74d8671fa0840bbfcb549ac; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66393145, jitterRate=-0.01066504418849945}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:26,554 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 51e4d6a6293430d21cdf75126c54f808: Running coprocessor pre-open hook at 1676088026540Writing region info on filesystem at 1676088026540Initializing all the Stores at 1676088026541 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088026541Cleaning up temporary data from old regions at 1676088026551 (+10 ms)Cleaning up detritus from prior splits at 1676088026551Running coprocessor post-open hooks at 1676088026554 (+3 ms)Region opened successfully at 1676088026554 2023-02-11 04:00:26,555 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for aaa51003a74d8671fa0840bbfcb549ac: Running coprocessor pre-open hook at 1676088026540Writing region info on filesystem at 1676088026540Initializing all the Stores at 1676088026541 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088026541Cleaning up temporary data from old regions at 1676088026551 (+10 ms)Cleaning up detritus from prior splits at 1676088026551Running coprocessor post-open hooks at 1676088026555 (+4 ms)Region opened successfully at 1676088026555 2023-02-11 04:00:26,555 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e1d0ef015d92b032f67c5052a11865c4; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67303773, jitterRate=0.002904370427131653}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:26,555 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e1d0ef015d92b032f67c5052a11865c4: Running coprocessor pre-open hook at 1676088026540Writing region info on filesystem at 1676088026540Initializing all the Stores at 1676088026541 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088026541Cleaning up temporary data from old regions at 1676088026551 (+10 ms)Cleaning up detritus from prior splits at 1676088026552 (+1 ms)Running coprocessor post-open hooks at 1676088026555 (+3 ms)Region opened successfully at 1676088026555 2023-02-11 04:00:26,556 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac., pid=103, masterSystemTime=1676088026535 2023-02-11 04:00:26,556 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4., pid=102, masterSystemTime=1676088026536 2023-02-11 04:00:26,556 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808., pid=104, masterSystemTime=1676088026536 2023-02-11 04:00:26,557 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:26,557 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:26,557 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:26,557 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d586dffdbc612899aa5c72d99bdc6519, NAME => 'testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 04:00:26,558 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:26,558 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:26,558 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:26,558 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:26,559 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=95 updating hbase:meta row=51e4d6a6293430d21cdf75126c54f808, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:26,559 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088026559"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088026559"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088026559"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088026559"}]},"ts":"1676088026559"} 2023-02-11 04:00:26,559 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:26,559 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:26,559 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:26,560 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 747a6fa324aa972cea6bbd307f3b1539, NAME => 'testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 04:00:26,560 INFO [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:26,560 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:26,560 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:26,560 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:26,560 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:26,560 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=96 updating hbase:meta row=e1d0ef015d92b032f67c5052a11865c4, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:26,560 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088026560"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088026560"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088026560"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088026560"}]},"ts":"1676088026560"} 2023-02-11 04:00:26,562 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=aaa51003a74d8671fa0840bbfcb549ac, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:26,562 INFO [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:26,563 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088026562"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088026562"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088026562"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088026562"}]},"ts":"1676088026562"} 2023-02-11 04:00:26,562 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:26,563 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:26,563 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:26,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 3452669615c81575c52ff96406a41e9e, NAME => 'testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 04:00:26,564 DEBUG [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf 2023-02-11 04:00:26,564 DEBUG [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf 2023-02-11 04:00:26,564 INFO [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d586dffdbc612899aa5c72d99bdc6519 columnFamilyName cf 2023-02-11 04:00:26,564 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:26,564 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:26,565 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:26,565 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:26,565 DEBUG [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf 2023-02-11 04:00:26,565 DEBUG [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf 2023-02-11 04:00:26,565 INFO [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 747a6fa324aa972cea6bbd307f3b1539 columnFamilyName cf 2023-02-11 04:00:26,566 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=104, resume processing ppid=95 2023-02-11 04:00:26,566 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=104, ppid=95, state=SUCCESS; OpenRegionProcedure 51e4d6a6293430d21cdf75126c54f808, server=jenkins-hbase4.apache.org,40829,1676088013739 in 172 msec 2023-02-11 04:00:26,566 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=102, resume processing ppid=96 2023-02-11 04:00:26,566 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=102, ppid=96, state=SUCCESS; OpenRegionProcedure e1d0ef015d92b032f67c5052a11865c4, server=jenkins-hbase4.apache.org,42385,1676088013771 in 178 msec 2023-02-11 04:00:26,568 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=95, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, ASSIGN in 342 msec 2023-02-11 04:00:26,568 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=103, resume processing ppid=97 2023-02-11 04:00:26,569 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=96, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, ASSIGN in 342 msec 2023-02-11 04:00:26,569 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=103, ppid=97, state=SUCCESS; OpenRegionProcedure aaa51003a74d8671fa0840bbfcb549ac, server=jenkins-hbase4.apache.org,46851,1676088013694 in 179 msec 2023-02-11 04:00:26,569 INFO [StoreOpener-3452669615c81575c52ff96406a41e9e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:26,570 DEBUG [StoreOpener-3452669615c81575c52ff96406a41e9e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf 2023-02-11 04:00:26,570 DEBUG [StoreOpener-3452669615c81575c52ff96406a41e9e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf 2023-02-11 04:00:26,570 INFO [StoreOpener-3452669615c81575c52ff96406a41e9e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3452669615c81575c52ff96406a41e9e columnFamilyName cf 2023-02-11 04:00:26,571 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=97, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, ASSIGN in 345 msec 2023-02-11 04:00:26,572 DEBUG [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/0a646c34d29c4963afc688a0da1fd6d4 2023-02-11 04:00:26,572 INFO [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] regionserver.HStore(310): Store=747a6fa324aa972cea6bbd307f3b1539/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:26,573 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:26,574 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:26,577 DEBUG [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/e928d1188fee412e9c504ba31bbb3ef6 2023-02-11 04:00:26,577 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:26,578 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 747a6fa324aa972cea6bbd307f3b1539; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67905395, jitterRate=0.011869236826896667}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:26,578 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 747a6fa324aa972cea6bbd307f3b1539: Running coprocessor pre-open hook at 1676088026560Writing region info on filesystem at 1676088026560Initializing all the Stores at 1676088026562 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088026562Cleaning up temporary data from old regions at 1676088026574 (+12 ms)Cleaning up detritus from prior splits at 1676088026575 (+1 ms)Running coprocessor post-open hooks at 1676088026578 (+3 ms)Region opened successfully at 1676088026578 2023-02-11 04:00:26,579 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539., pid=101, masterSystemTime=1676088026536 2023-02-11 04:00:26,580 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:26,580 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:26,581 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=747a6fa324aa972cea6bbd307f3b1539, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:26,581 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088026581"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088026581"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088026581"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088026581"}]},"ts":"1676088026581"} 2023-02-11 04:00:26,584 INFO [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] regionserver.HStore(310): Store=d586dffdbc612899aa5c72d99bdc6519/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:26,585 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=101, resume processing ppid=93 2023-02-11 04:00:26,585 DEBUG [StoreOpener-3452669615c81575c52ff96406a41e9e-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/a11c828e49bd4e7a9cf05be9ff07ad7f 2023-02-11 04:00:26,585 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:26,585 INFO [StoreOpener-3452669615c81575c52ff96406a41e9e-1] regionserver.HStore(310): Store=3452669615c81575c52ff96406a41e9e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:26,585 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=101, ppid=93, state=SUCCESS; OpenRegionProcedure 747a6fa324aa972cea6bbd307f3b1539, server=jenkins-hbase4.apache.org,42385,1676088013771 in 199 msec 2023-02-11 04:00:26,587 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:26,587 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:26,587 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=93, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, ASSIGN in 363 msec 2023-02-11 04:00:26,588 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:26,590 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:26,590 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:26,590 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened d586dffdbc612899aa5c72d99bdc6519; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59445978, jitterRate=-0.11418589949607849}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:26,591 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for d586dffdbc612899aa5c72d99bdc6519: Running coprocessor pre-open hook at 1676088026558Writing region info on filesystem at 1676088026558Initializing all the Stores at 1676088026559 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088026559Cleaning up temporary data from old regions at 1676088026587 (+28 ms)Cleaning up detritus from prior splits at 1676088026588 (+1 ms)Running coprocessor post-open hooks at 1676088026591 (+3 ms)Region opened successfully at 1676088026591 2023-02-11 04:00:26,591 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519., pid=100, masterSystemTime=1676088026536 2023-02-11 04:00:26,591 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 3452669615c81575c52ff96406a41e9e; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61241097, jitterRate=-0.08743654191493988}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:26,591 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 3452669615c81575c52ff96406a41e9e: Running coprocessor pre-open hook at 1676088026566Writing region info on filesystem at 1676088026566Initializing all the Stores at 1676088026568 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088026568Cleaning up temporary data from old regions at 1676088026588 (+20 ms)Cleaning up detritus from prior splits at 1676088026589 (+1 ms)Running coprocessor post-open hooks at 1676088026591 (+2 ms)Region opened successfully at 1676088026591 2023-02-11 04:00:26,593 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e., pid=99, masterSystemTime=1676088026535 2023-02-11 04:00:26,594 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:26,594 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:26,595 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=d586dffdbc612899aa5c72d99bdc6519, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:26,595 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:26,595 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088026594"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088026594"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088026594"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088026594"}]},"ts":"1676088026594"} 2023-02-11 04:00:26,595 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:26,595 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=94 updating hbase:meta row=3452669615c81575c52ff96406a41e9e, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:26,596 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088026595"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088026595"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088026595"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088026595"}]},"ts":"1676088026595"} 2023-02-11 04:00:26,598 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=100, resume processing ppid=98 2023-02-11 04:00:26,598 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=100, ppid=98, state=SUCCESS; OpenRegionProcedure d586dffdbc612899aa5c72d99bdc6519, server=jenkins-hbase4.apache.org,40829,1676088013739 in 213 msec 2023-02-11 04:00:26,599 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=99, resume processing ppid=94 2023-02-11 04:00:26,599 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=99, ppid=94, state=SUCCESS; OpenRegionProcedure 3452669615c81575c52ff96406a41e9e, server=jenkins-hbase4.apache.org,46851,1676088013694 in 214 msec 2023-02-11 04:00:26,600 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=98, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, ASSIGN in 375 msec 2023-02-11 04:00:26,601 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=94, resume processing ppid=92 2023-02-11 04:00:26,601 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=94, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, ASSIGN in 377 msec 2023-02-11 04:00:26,601 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088026601"}]},"ts":"1676088026601"} 2023-02-11 04:00:26,603 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676088016466, state=ENABLED in hbase:meta 2023-02-11 04:00:26,604 INFO [PEWorker-1] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676088016466' was successfully enabled. 2023-02-11 04:00:26,606 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=92, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676088016466 in 392 msec 2023-02-11 04:00:26,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-11 04:00:26,968 INFO [Listener at localhost/33753] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676088016466, procId: 92 completed 2023-02-11 04:00:26,969 DEBUG [Listener at localhost/33753] zookeeper.ReadOnlyZKClient(139): Connect 0x251148a9 to 127.0.0.1:63425 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 04:00:26,975 DEBUG [Listener at localhost/33753] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7c1707f4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 04:00:26,976 DEBUG [hconnection-0x29e7c4cf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 04:00:26,983 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47752, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 04:00:26,989 DEBUG [hconnection-0x29e7c4cf-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 04:00:26,991 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34990, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 04:00:27,064 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 04:00:27,067 DEBUG [hconnection-0x29e7c4cf-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 04:00:27,071 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37110, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 04:00:27,151 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 04:00:27,257 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 04:00:27,362 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 04:00:27,435 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 04:00:28,380 INFO [Listener at localhost/33753] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676088016466 2023-02-11 04:00:28,380 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676088016466 2023-02-11 04:00:28,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] procedure2.ProcedureExecutor(1029): Stored pid=105, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:28,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-11 04:00:28,384 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088028384"}]},"ts":"1676088028384"} 2023-02-11 04:00:28,385 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676088016466, state=DISABLING in hbase:meta 2023-02-11 04:00:28,387 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676088016466 to state=DISABLING 2023-02-11 04:00:28,388 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, UNASSIGN}, {pid=107, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, UNASSIGN}, {pid=108, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, UNASSIGN}, {pid=109, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, UNASSIGN}, {pid=110, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, UNASSIGN}, {pid=111, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, UNASSIGN}] 2023-02-11 04:00:28,389 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=111, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, UNASSIGN 2023-02-11 04:00:28,390 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=107, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, UNASSIGN 2023-02-11 04:00:28,390 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=110, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, UNASSIGN 2023-02-11 04:00:28,390 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, UNASSIGN 2023-02-11 04:00:28,390 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=109, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, UNASSIGN 2023-02-11 04:00:28,391 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=111 updating hbase:meta row=d586dffdbc612899aa5c72d99bdc6519, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:28,391 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=aaa51003a74d8671fa0840bbfcb549ac, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:28,391 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=107 updating hbase:meta row=3452669615c81575c52ff96406a41e9e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:28,391 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088028391"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088028391"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088028391"}]},"ts":"1676088028391"} 2023-02-11 04:00:28,391 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088028391"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088028391"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088028391"}]},"ts":"1676088028391"} 2023-02-11 04:00:28,391 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=747a6fa324aa972cea6bbd307f3b1539, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:28,391 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=109 updating hbase:meta row=e1d0ef015d92b032f67c5052a11865c4, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:28,391 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088028391"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088028391"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088028391"}]},"ts":"1676088028391"} 2023-02-11 04:00:28,391 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088028391"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088028391"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088028391"}]},"ts":"1676088028391"} 2023-02-11 04:00:28,391 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088028391"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088028391"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088028391"}]},"ts":"1676088028391"} 2023-02-11 04:00:28,393 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=112, ppid=110, state=RUNNABLE; CloseRegionProcedure aaa51003a74d8671fa0840bbfcb549ac, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:28,396 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=113, ppid=111, state=RUNNABLE; CloseRegionProcedure d586dffdbc612899aa5c72d99bdc6519, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:28,397 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=114, ppid=106, state=RUNNABLE; CloseRegionProcedure 747a6fa324aa972cea6bbd307f3b1539, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:28,398 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=115, ppid=109, state=RUNNABLE; CloseRegionProcedure e1d0ef015d92b032f67c5052a11865c4, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:28,399 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=116, ppid=107, state=RUNNABLE; CloseRegionProcedure 3452669615c81575c52ff96406a41e9e, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:28,399 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=108, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, UNASSIGN 2023-02-11 04:00:28,401 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=108 updating hbase:meta row=51e4d6a6293430d21cdf75126c54f808, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:28,401 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088028401"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088028401"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088028401"}]},"ts":"1676088028401"} 2023-02-11 04:00:28,403 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=117, ppid=108, state=RUNNABLE; CloseRegionProcedure 51e4d6a6293430d21cdf75126c54f808, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:28,547 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:28,549 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing aaa51003a74d8671fa0840bbfcb549ac, disabling compactions & flushes 2023-02-11 04:00:28,549 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:28,549 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:28,549 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. after waiting 0 ms 2023-02-11 04:00:28,549 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:28,549 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:28,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 51e4d6a6293430d21cdf75126c54f808, disabling compactions & flushes 2023-02-11 04:00:28,551 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:28,552 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:28,553 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. after waiting 0 ms 2023-02-11 04:00:28,553 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:28,553 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:28,553 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e1d0ef015d92b032f67c5052a11865c4, disabling compactions & flushes 2023-02-11 04:00:28,553 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:28,553 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:28,553 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. after waiting 0 ms 2023-02-11 04:00:28,554 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:28,561 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-11 04:00:28,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-11 04:00:28,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-11 04:00:28,562 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:28,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for aaa51003a74d8671fa0840bbfcb549ac: Waiting for close lock at 1676088028548Running coprocessor pre-close hooks at 1676088028548Disabling compacts and flushes for region at 1676088028548Disabling writes for close at 1676088028549 (+1 ms)Writing region close event to WAL at 1676088028553 (+4 ms)Running coprocessor post-close hooks at 1676088028562 (+9 ms)Closed at 1676088028562 2023-02-11 04:00:28,562 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:28,563 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 51e4d6a6293430d21cdf75126c54f808: Waiting for close lock at 1676088028549Running coprocessor pre-close hooks at 1676088028549Disabling compacts and flushes for region at 1676088028549Disabling writes for close at 1676088028553 (+4 ms)Writing region close event to WAL at 1676088028559 (+6 ms)Running coprocessor post-close hooks at 1676088028562 (+3 ms)Closed at 1676088028562 2023-02-11 04:00:28,563 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:28,563 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e1d0ef015d92b032f67c5052a11865c4: Waiting for close lock at 1676088028553Running coprocessor pre-close hooks at 1676088028553Disabling compacts and flushes for region at 1676088028553Disabling writes for close at 1676088028553Writing region close event to WAL at 1676088028559 (+6 ms)Running coprocessor post-close hooks at 1676088028562 (+3 ms)Closed at 1676088028563 (+1 ms) 2023-02-11 04:00:28,563 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:28,564 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:28,564 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 3452669615c81575c52ff96406a41e9e, disabling compactions & flushes 2023-02-11 04:00:28,564 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:28,564 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:28,564 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. after waiting 0 ms 2023-02-11 04:00:28,564 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:28,564 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=aaa51003a74d8671fa0840bbfcb549ac, regionState=CLOSED 2023-02-11 04:00:28,564 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088028564"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088028564"}]},"ts":"1676088028564"} 2023-02-11 04:00:28,564 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:28,564 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:28,565 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing d586dffdbc612899aa5c72d99bdc6519, disabling compactions & flushes 2023-02-11 04:00:28,565 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:28,565 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=108 updating hbase:meta row=51e4d6a6293430d21cdf75126c54f808, regionState=CLOSED 2023-02-11 04:00:28,565 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:28,565 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. after waiting 0 ms 2023-02-11 04:00:28,565 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088028565"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088028565"}]},"ts":"1676088028565"} 2023-02-11 04:00:28,565 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:28,565 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:28,565 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:28,565 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 747a6fa324aa972cea6bbd307f3b1539, disabling compactions & flushes 2023-02-11 04:00:28,565 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:28,566 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:28,566 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. after waiting 0 ms 2023-02-11 04:00:28,566 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:28,566 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=109 updating hbase:meta row=e1d0ef015d92b032f67c5052a11865c4, regionState=CLOSED 2023-02-11 04:00:28,567 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088028566"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088028566"}]},"ts":"1676088028566"} 2023-02-11 04:00:28,569 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=112, resume processing ppid=110 2023-02-11 04:00:28,569 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=112, ppid=110, state=SUCCESS; CloseRegionProcedure aaa51003a74d8671fa0840bbfcb549ac, server=jenkins-hbase4.apache.org,46851,1676088013694 in 173 msec 2023-02-11 04:00:28,570 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=117, resume processing ppid=108 2023-02-11 04:00:28,570 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=117, ppid=108, state=SUCCESS; CloseRegionProcedure 51e4d6a6293430d21cdf75126c54f808, server=jenkins-hbase4.apache.org,40829,1676088013739 in 164 msec 2023-02-11 04:00:28,570 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=110, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, UNASSIGN in 181 msec 2023-02-11 04:00:28,570 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=115, resume processing ppid=109 2023-02-11 04:00:28,570 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=115, ppid=109, state=SUCCESS; CloseRegionProcedure e1d0ef015d92b032f67c5052a11865c4, server=jenkins-hbase4.apache.org,42385,1676088013771 in 170 msec 2023-02-11 04:00:28,571 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-11 04:00:28,571 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=108, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, UNASSIGN in 182 msec 2023-02-11 04:00:28,572 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:28,572 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 3452669615c81575c52ff96406a41e9e: Waiting for close lock at 1676088028564Running coprocessor pre-close hooks at 1676088028564Disabling compacts and flushes for region at 1676088028564Disabling writes for close at 1676088028564Writing region close event to WAL at 1676088028565 (+1 ms)Running coprocessor post-close hooks at 1676088028572 (+7 ms)Closed at 1676088028572 2023-02-11 04:00:28,572 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=109, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, UNASSIGN in 182 msec 2023-02-11 04:00:28,572 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-11 04:00:28,572 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-11 04:00:28,573 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:28,573 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:28,573 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 747a6fa324aa972cea6bbd307f3b1539: Waiting for close lock at 1676088028565Running coprocessor pre-close hooks at 1676088028565Disabling compacts and flushes for region at 1676088028565Disabling writes for close at 1676088028566 (+1 ms)Writing region close event to WAL at 1676088028567 (+1 ms)Running coprocessor post-close hooks at 1676088028573 (+6 ms)Closed at 1676088028573 2023-02-11 04:00:28,573 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for d586dffdbc612899aa5c72d99bdc6519: Waiting for close lock at 1676088028564Running coprocessor pre-close hooks at 1676088028564Disabling compacts and flushes for region at 1676088028564Disabling writes for close at 1676088028565 (+1 ms)Writing region close event to WAL at 1676088028566 (+1 ms)Running coprocessor post-close hooks at 1676088028573 (+7 ms)Closed at 1676088028573 2023-02-11 04:00:28,573 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:28,574 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=107 updating hbase:meta row=3452669615c81575c52ff96406a41e9e, regionState=CLOSED 2023-02-11 04:00:28,574 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088028574"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088028574"}]},"ts":"1676088028574"} 2023-02-11 04:00:28,574 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:28,575 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=747a6fa324aa972cea6bbd307f3b1539, regionState=CLOSED 2023-02-11 04:00:28,575 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088028574"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088028574"}]},"ts":"1676088028574"} 2023-02-11 04:00:28,575 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:28,576 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=111 updating hbase:meta row=d586dffdbc612899aa5c72d99bdc6519, regionState=CLOSED 2023-02-11 04:00:28,576 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088028576"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088028576"}]},"ts":"1676088028576"} 2023-02-11 04:00:28,578 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=116, resume processing ppid=107 2023-02-11 04:00:28,578 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=116, ppid=107, state=SUCCESS; CloseRegionProcedure 3452669615c81575c52ff96406a41e9e, server=jenkins-hbase4.apache.org,46851,1676088013694 in 176 msec 2023-02-11 04:00:28,579 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=114, resume processing ppid=106 2023-02-11 04:00:28,579 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=113, resume processing ppid=111 2023-02-11 04:00:28,579 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=114, ppid=106, state=SUCCESS; CloseRegionProcedure 747a6fa324aa972cea6bbd307f3b1539, server=jenkins-hbase4.apache.org,42385,1676088013771 in 179 msec 2023-02-11 04:00:28,579 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=113, ppid=111, state=SUCCESS; CloseRegionProcedure d586dffdbc612899aa5c72d99bdc6519, server=jenkins-hbase4.apache.org,40829,1676088013739 in 183 msec 2023-02-11 04:00:28,580 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=107, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, UNASSIGN in 190 msec 2023-02-11 04:00:28,580 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=106, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, UNASSIGN in 191 msec 2023-02-11 04:00:28,581 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=111, resume processing ppid=105 2023-02-11 04:00:28,581 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=111, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, UNASSIGN in 191 msec 2023-02-11 04:00:28,582 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088028582"}]},"ts":"1676088028582"} 2023-02-11 04:00:28,583 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676088016466, state=DISABLED in hbase:meta 2023-02-11 04:00:28,590 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676088016466 to state=DISABLED 2023-02-11 04:00:28,591 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=105, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676088016466 in 210 msec 2023-02-11 04:00:28,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-11 04:00:28,635 INFO [Listener at localhost/33753] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676088016466, procId: 105 completed 2023-02-11 04:00:28,644 INFO [Listener at localhost/33753] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-emptySnaptb-1676088016466-1676088028644 2023-02-11 04:00:28,645 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=hbase-failsafe-emptySnaptb-1676088016466-1676088028644 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 } 2023-02-11 04:00:28,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676088028645 (current time:1676088028645). 2023-02-11 04:00:28,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-11 04:00:28,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] zookeeper.ReadOnlyZKClient(139): Connect 0x073b975c to 127.0.0.1:63425 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 04:00:28,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1cd338cf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 04:00:28,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 04:00:28,654 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47758, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 04:00:28,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x073b975c to 127.0.0.1:63425 2023-02-11 04:00:28,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:00:28,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-11 04:00:28,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-emptySnaptb-1676088016466-1676088028644 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 } 2023-02-11 04:00:28,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] procedure2.ProcedureExecutor(1029): Stored pid=118, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676088016466, type=EXCLUSIVE 2023-02-11 04:00:28,658 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=118, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676088016466, type=EXCLUSIVE 2023-02-11 04:00:28,658 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=118, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676088016466, type=EXCLUSIVE; timeout=600000, timestamp=1676088628658 2023-02-11 04:00:28,658 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-emptySnaptb-1676088016466-1676088028644 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676088016466 2023-02-11 04:00:28,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-emptySnaptb-1676088016466-1676088028644 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 } 2023-02-11 04:00:28,659 DEBUG [Listener at localhost/33753] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-emptySnaptb-1676088016466-1676088028644 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-11 04:00:28,659 DEBUG [Listener at localhost/33753] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-11 04:00:28,668 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-emptySnaptb-1676088016466-1676088028644 table=testRestoreSnapshot-1676088016466 type=DISABLED ttl=0 } 2023-02-11 04:00:28,670 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:28,670 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:28,670 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:28,670 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:28,670 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:28,671 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:28,671 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:28,671 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:28,671 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:28,671 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:28,671 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:28,671 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:28,671 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:28,671 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:28,672 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/93d3b4e383ae4310a66075c83ee5c664] hfiles 2023-02-11 04:00:28,672 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/93d3b4e383ae4310a66075c83ee5c664 2023-02-11 04:00:28,673 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/e928d1188fee412e9c504ba31bbb3ef6] hfiles 2023-02-11 04:00:28,673 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/e928d1188fee412e9c504ba31bbb3ef6 2023-02-11 04:00:28,673 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/a11c828e49bd4e7a9cf05be9ff07ad7f] hfiles 2023-02-11 04:00:28,673 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/a11c828e49bd4e7a9cf05be9ff07ad7f 2023-02-11 04:00:28,673 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/1b38064ae3304eca9c176d5669359823] hfiles 2023-02-11 04:00:28,673 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/1b38064ae3304eca9c176d5669359823 2023-02-11 04:00:28,678 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45] hfiles 2023-02-11 04:00:28,678 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 2023-02-11 04:00:28,678 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 2023-02-11 04:00:28,678 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 2023-02-11 04:00:28,678 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 2023-02-11 04:00:28,678 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 2023-02-11 04:00:28,678 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 2023-02-11 04:00:28,679 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/3f27b1f3105b45bf81c17ba9c704bf44] hfiles 2023-02-11 04:00:28,679 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/3f27b1f3105b45bf81c17ba9c704bf44 2023-02-11 04:00:28,680 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/0a646c34d29c4963afc688a0da1fd6d4] hfiles 2023-02-11 04:00:28,680 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/0a646c34d29c4963afc688a0da1fd6d4 2023-02-11 04:00:28,708 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-emptySnaptb-1676088016466-1676088028644 table=testRestoreSnapshot-1676088016466 type=DISABLED ttl=0 } as finished. 2023-02-11 04:00:28,708 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-emptySnaptb-1676088016466-1676088028644 2023-02-11 04:00:28,708 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676088016466-1676088028644 2023-02-11 04:00:28,733 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676088016466-1676088028644 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/hbase-failsafe-emptySnaptb-1676088016466-1676088028644 2023-02-11 04:00:28,752 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-emptySnaptb-1676088016466-1676088028644 of table testRestoreSnapshot-1676088016466 completed 2023-02-11 04:00:28,752 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676088016466-1676088028644 2023-02-11 04:00:28,753 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676088016466-1676088028644 2023-02-11 04:00:28,753 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-emptySnaptb-1676088016466-1676088028644 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676088016466 at 1676088028658Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-emptySnaptb-1676088016466-1676088028644 table=testRestoreSnapshot-1676088016466 type=DISABLED ttl=0 } at 1676088028668 (+10 ms)Consolidate snapshot: hbase-failsafe-emptySnaptb-1676088016466-1676088028644 at 1676088028708 (+40 ms)Loading Region manifests for hbase-failsafe-emptySnaptb-1676088016466-1676088028644 at 1676088028708Writing data manifest for hbase-failsafe-emptySnaptb-1676088016466-1676088028644 at 1676088028716 (+8 ms)Verifying snapshot: hbase-failsafe-emptySnaptb-1676088016466-1676088028644 at 1676088028726 (+10 ms)Snapshot hbase-failsafe-emptySnaptb-1676088016466-1676088028644 of table testRestoreSnapshot-1676088016466 completed at 1676088028752 (+26 ms) 2023-02-11 04:00:28,755 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=118, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676088016466, type=EXCLUSIVE 2023-02-11 04:00:28,758 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=118, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676088016466, type=EXCLUSIVE in 100 msec 2023-02-11 04:00:28,909 DEBUG [Listener at localhost/33753] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 04:00:28,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-emptySnaptb-1676088016466-1676088028644 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 } is done 2023-02-11 04:00:28,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-emptySnaptb-1676088016466-1676088028644 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-11 04:00:28,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] procedure2.ProcedureExecutor(1029): Stored pid=119, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676088016466 snapshot=name: "emptySnaptb-1676088016466" table: "testRestoreSnapshot-1676088016466" creation_time: 1676088017599 type: DISABLED version: 2 ttl: 0 ) 2023-02-11 04:00:28,917 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(880): Restore snapshot=emptySnaptb-1676088016466 as table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:28,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-11 04:00:28,929 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/.tabledesc/.tableinfo.0000000003 2023-02-11 04:00:28,929 DEBUG [PEWorker-5] util.FSTableDescriptors(519): Deleted hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/.tabledesc/.tableinfo.0000000002 2023-02-11 04:00:28,929 INFO [PEWorker-5] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/.tabledesc/.tableinfo.0000000003 2023-02-11 04:00:28,930 INFO [PEWorker-5] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=emptySnaptb-1676088016466 table=testRestoreSnapshot-1676088016466 type=DISABLED ttl=0 } 2023-02-11 04:00:28,931 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "emptySnaptb-1676088016466" table: "testRestoreSnapshot-1676088016466" creation_time: 1676088017599 type: DISABLED version: 2 ttl: 0 2023-02-11 04:00:28,931 DEBUG [PEWorker-5] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466 2023-02-11 04:00:28,938 DEBUG [PEWorker-5] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:28,941 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 2023-02-11 04:00:28,942 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 2023-02-11 04:00:28,943 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 2023-02-11 04:00:28,944 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 2023-02-11 04:00:28,946 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 2023-02-11 04:00:28,947 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 2023-02-11 04:00:28,947 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:28,947 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:28,947 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:28,947 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:28,947 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:28,947 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:28,952 DEBUG [RestoreSnapshot-pool-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/1b38064ae3304eca9c176d5669359823 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/1b38064ae3304eca9c176d5669359823 2023-02-11 04:00:28,952 DEBUG [RestoreSnapshot-pool-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/a11c828e49bd4e7a9cf05be9ff07ad7f to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/a11c828e49bd4e7a9cf05be9ff07ad7f 2023-02-11 04:00:28,952 DEBUG [RestoreSnapshot-pool-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/0a646c34d29c4963afc688a0da1fd6d4 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/0a646c34d29c4963afc688a0da1fd6d4 2023-02-11 04:00:28,953 DEBUG [RestoreSnapshot-pool-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/93d3b4e383ae4310a66075c83ee5c664 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/93d3b4e383ae4310a66075c83ee5c664 2023-02-11 04:00:28,953 DEBUG [RestoreSnapshot-pool-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/3f27b1f3105b45bf81c17ba9c704bf44 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/3f27b1f3105b45bf81c17ba9c704bf44 2023-02-11 04:00:28,954 DEBUG [RestoreSnapshot-pool-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/e928d1188fee412e9c504ba31bbb3ef6 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/e928d1188fee412e9c504ba31bbb3ef6 2023-02-11 04:00:28,954 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "emptySnaptb-1676088016466" table: "testRestoreSnapshot-1676088016466" creation_time: 1676088017599 type: DISABLED version: 2 ttl: 0 2023-02-11 04:00:28,955 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088028955"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:28,955 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088028955"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:28,955 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088028955"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:28,955 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088028955"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:28,956 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088028955"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:28,956 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088028955"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:28,958 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-11 04:00:28,958 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 3452669615c81575c52ff96406a41e9e, NAME => 'testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 51e4d6a6293430d21cdf75126c54f808, NAME => 'testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 747a6fa324aa972cea6bbd307f3b1539, NAME => 'testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => aaa51003a74d8671fa0840bbfcb549ac, NAME => 'testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => d586dffdbc612899aa5c72d99bdc6519, NAME => 'testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => e1d0ef015d92b032f67c5052a11865c4, NAME => 'testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.', STARTKEY => '3', ENDKEY => '4'}] 2023-02-11 04:00:28,958 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088028956"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088028956"}]},"ts":"1676088028956"} 2023-02-11 04:00:28,958 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088028956"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088028956"}]},"ts":"1676088028956"} 2023-02-11 04:00:28,958 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088028956"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088028956"}]},"ts":"1676088028956"} 2023-02-11 04:00:28,958 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088028956"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088028956"}]},"ts":"1676088028956"} 2023-02-11 04:00:28,958 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088028956"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088028956"}]},"ts":"1676088028956"} 2023-02-11 04:00:28,958 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088028956"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088028956"}]},"ts":"1676088028956"} 2023-02-11 04:00:28,960 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-11 04:00:28,960 INFO [PEWorker-5] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-11 04:00:28,960 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 3452669615c81575c52ff96406a41e9e, NAME => 'testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 51e4d6a6293430d21cdf75126c54f808, NAME => 'testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 747a6fa324aa972cea6bbd307f3b1539, NAME => 'testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => aaa51003a74d8671fa0840bbfcb549ac, NAME => 'testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => d586dffdbc612899aa5c72d99bdc6519, NAME => 'testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => e1d0ef015d92b032f67c5052a11865c4, NAME => 'testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.', STARTKEY => '3', ENDKEY => '4'}] 2023-02-11 04:00:28,960 INFO [PEWorker-5] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=emptySnaptb-1676088016466 table=testRestoreSnapshot-1676088016466 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676088016466 completed! 2023-02-11 04:00:28,963 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=119, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676088016466 snapshot=name: "emptySnaptb-1676088016466" table: "testRestoreSnapshot-1676088016466" creation_time: 1676088017599 type: DISABLED version: 2 ttl: 0 ) in 45 msec 2023-02-11 04:00:29,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-11 04:00:29,170 INFO [Listener at localhost/33753] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676088016466, procId: 119 completed 2023-02-11 04:00:29,170 INFO [Listener at localhost/33753] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-emptySnaptb-1676088016466-1676088028644 2023-02-11 04:00:29,171 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "hbase-failsafe-emptySnaptb-1676088016466-1676088028644" 2023-02-11 04:00:29,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-emptySnaptb-1676088016466-1676088028644 2023-02-11 04:00:29,174 INFO [Listener at localhost/33753] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676088016466 2023-02-11 04:00:29,174 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676088016466 2023-02-11 04:00:29,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] procedure2.ProcedureExecutor(1029): Stored pid=120, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:29,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-11 04:00:29,177 INFO [PEWorker-4] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676088016466 2023-02-11 04:00:29,177 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088029177"}]},"ts":"1676088029177"} 2023-02-11 04:00:29,178 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676088016466, state=ENABLING in hbase:meta 2023-02-11 04:00:29,181 DEBUG [PEWorker-4] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-11 04:00:29,181 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, ASSIGN}, {pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, ASSIGN}, {pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, ASSIGN}, {pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, ASSIGN}, {pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, ASSIGN}, {pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, ASSIGN}] 2023-02-11 04:00:29,183 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, ASSIGN 2023-02-11 04:00:29,183 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, ASSIGN 2023-02-11 04:00:29,183 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, ASSIGN 2023-02-11 04:00:29,183 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, ASSIGN 2023-02-11 04:00:29,183 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 04:00:29,184 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 04:00:29,184 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 04:00:29,184 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, ASSIGN 2023-02-11 04:00:29,184 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 04:00:29,185 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, ASSIGN 2023-02-11 04:00:29,185 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 04:00:29,185 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 04:00:29,334 DEBUG [jenkins-hbase4:41713] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 04:00:29,334 DEBUG [jenkins-hbase4:41713] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 04:00:29,334 DEBUG [jenkins-hbase4:41713] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 04:00:29,334 DEBUG [jenkins-hbase4:41713] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 04:00:29,334 DEBUG [jenkins-hbase4:41713] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 04:00:29,339 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=d586dffdbc612899aa5c72d99bdc6519, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:29,339 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=122 updating hbase:meta row=3452669615c81575c52ff96406a41e9e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:29,339 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=125 updating hbase:meta row=aaa51003a74d8671fa0840bbfcb549ac, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:29,339 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088029339"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088029339"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088029339"}]},"ts":"1676088029339"} 2023-02-11 04:00:29,339 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088029339"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088029339"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088029339"}]},"ts":"1676088029339"} 2023-02-11 04:00:29,339 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=123 updating hbase:meta row=51e4d6a6293430d21cdf75126c54f808, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:29,339 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=124 updating hbase:meta row=e1d0ef015d92b032f67c5052a11865c4, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:29,339 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088029339"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088029339"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088029339"}]},"ts":"1676088029339"} 2023-02-11 04:00:29,339 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088029339"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088029339"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088029339"}]},"ts":"1676088029339"} 2023-02-11 04:00:29,339 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088029339"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088029339"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088029339"}]},"ts":"1676088029339"} 2023-02-11 04:00:29,340 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=127, ppid=126, state=RUNNABLE; OpenRegionProcedure d586dffdbc612899aa5c72d99bdc6519, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:29,341 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=128, ppid=122, state=RUNNABLE; OpenRegionProcedure 3452669615c81575c52ff96406a41e9e, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:29,342 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=129, ppid=125, state=RUNNABLE; OpenRegionProcedure aaa51003a74d8671fa0840bbfcb549ac, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:29,343 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=130, ppid=124, state=RUNNABLE; OpenRegionProcedure e1d0ef015d92b032f67c5052a11865c4, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:29,344 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=131, ppid=123, state=RUNNABLE; OpenRegionProcedure 51e4d6a6293430d21cdf75126c54f808, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:29,344 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=747a6fa324aa972cea6bbd307f3b1539, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:29,344 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088029344"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088029344"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088029344"}]},"ts":"1676088029344"} 2023-02-11 04:00:29,346 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=132, ppid=121, state=RUNNABLE; OpenRegionProcedure 747a6fa324aa972cea6bbd307f3b1539, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:29,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-11 04:00:29,495 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:29,495 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d586dffdbc612899aa5c72d99bdc6519, NAME => 'testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 04:00:29,495 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:29,495 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:29,495 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 3452669615c81575c52ff96406a41e9e, NAME => 'testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 04:00:29,495 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:29,495 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:29,495 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:29,496 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:29,496 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:29,496 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:29,496 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:29,497 INFO [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:29,497 INFO [StoreOpener-3452669615c81575c52ff96406a41e9e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:29,497 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:29,497 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e1d0ef015d92b032f67c5052a11865c4, NAME => 'testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 04:00:29,497 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:29,498 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:29,498 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:29,498 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:29,498 DEBUG [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf 2023-02-11 04:00:29,498 DEBUG [StoreOpener-3452669615c81575c52ff96406a41e9e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf 2023-02-11 04:00:29,498 DEBUG [StoreOpener-3452669615c81575c52ff96406a41e9e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf 2023-02-11 04:00:29,498 DEBUG [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf 2023-02-11 04:00:29,498 INFO [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d586dffdbc612899aa5c72d99bdc6519 columnFamilyName cf 2023-02-11 04:00:29,498 INFO [StoreOpener-3452669615c81575c52ff96406a41e9e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3452669615c81575c52ff96406a41e9e columnFamilyName cf 2023-02-11 04:00:29,499 INFO [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:29,500 INFO [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] regionserver.HStore(310): Store=d586dffdbc612899aa5c72d99bdc6519/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:29,500 INFO [StoreOpener-3452669615c81575c52ff96406a41e9e-1] regionserver.HStore(310): Store=3452669615c81575c52ff96406a41e9e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:29,500 DEBUG [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf 2023-02-11 04:00:29,500 DEBUG [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf 2023-02-11 04:00:29,500 INFO [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e1d0ef015d92b032f67c5052a11865c4 columnFamilyName cf 2023-02-11 04:00:29,500 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:29,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:29,501 INFO [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] regionserver.HStore(310): Store=e1d0ef015d92b032f67c5052a11865c4/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:29,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:29,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:29,502 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:29,503 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:29,503 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:29,504 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:29,504 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened d586dffdbc612899aa5c72d99bdc6519; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73534707, jitterRate=0.0957525223493576}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:29,504 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for d586dffdbc612899aa5c72d99bdc6519: Running coprocessor pre-open hook at 1676088029496Writing region info on filesystem at 1676088029496Initializing all the Stores at 1676088029496Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088029496Cleaning up temporary data from old regions at 1676088029502 (+6 ms)Cleaning up detritus from prior splits at 1676088029502Running coprocessor post-open hooks at 1676088029504 (+2 ms)Region opened successfully at 1676088029504 2023-02-11 04:00:29,505 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 3452669615c81575c52ff96406a41e9e; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68114505, jitterRate=0.014985218644142151}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:29,505 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 3452669615c81575c52ff96406a41e9e: Running coprocessor pre-open hook at 1676088029496Writing region info on filesystem at 1676088029496Initializing all the Stores at 1676088029496Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088029496Cleaning up temporary data from old regions at 1676088029502 (+6 ms)Cleaning up detritus from prior splits at 1676088029502Running coprocessor post-open hooks at 1676088029505 (+3 ms)Region opened successfully at 1676088029505 2023-02-11 04:00:29,505 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519., pid=127, masterSystemTime=1676088029492 2023-02-11 04:00:29,505 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:29,505 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e., pid=128, masterSystemTime=1676088029493 2023-02-11 04:00:29,506 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e1d0ef015d92b032f67c5052a11865c4; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71370780, jitterRate=0.06350749731063843}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:29,506 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e1d0ef015d92b032f67c5052a11865c4: Running coprocessor pre-open hook at 1676088029498Writing region info on filesystem at 1676088029498Initializing all the Stores at 1676088029498Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088029498Cleaning up temporary data from old regions at 1676088029503 (+5 ms)Cleaning up detritus from prior splits at 1676088029503Running coprocessor post-open hooks at 1676088029506 (+3 ms)Region opened successfully at 1676088029506 2023-02-11 04:00:29,506 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:29,506 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:29,507 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:29,507 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 51e4d6a6293430d21cdf75126c54f808, NAME => 'testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 04:00:29,507 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4., pid=130, masterSystemTime=1676088029495 2023-02-11 04:00:29,507 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:29,507 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:29,507 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:29,507 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=d586dffdbc612899aa5c72d99bdc6519, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:29,507 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:29,507 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:29,507 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:29,507 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088029507"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088029507"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088029507"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088029507"}]},"ts":"1676088029507"} 2023-02-11 04:00:29,507 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:29,507 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => aaa51003a74d8671fa0840bbfcb549ac, NAME => 'testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 04:00:29,508 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:29,508 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:29,508 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:29,508 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:29,509 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=122 updating hbase:meta row=3452669615c81575c52ff96406a41e9e, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:29,509 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088029509"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088029509"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088029509"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088029509"}]},"ts":"1676088029509"} 2023-02-11 04:00:29,509 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:29,509 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:29,509 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:29,509 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 747a6fa324aa972cea6bbd307f3b1539, NAME => 'testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 04:00:29,509 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:29,509 INFO [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:29,510 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:29,510 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:29,510 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:29,510 INFO [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:29,511 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=124 updating hbase:meta row=e1d0ef015d92b032f67c5052a11865c4, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:29,511 INFO [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:29,511 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088029511"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088029511"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088029511"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088029511"}]},"ts":"1676088029511"} 2023-02-11 04:00:29,511 DEBUG [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf 2023-02-11 04:00:29,511 DEBUG [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf 2023-02-11 04:00:29,512 INFO [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 51e4d6a6293430d21cdf75126c54f808 columnFamilyName cf 2023-02-11 04:00:29,512 DEBUG [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf 2023-02-11 04:00:29,512 DEBUG [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf 2023-02-11 04:00:29,512 INFO [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region aaa51003a74d8671fa0840bbfcb549ac columnFamilyName cf 2023-02-11 04:00:29,513 INFO [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] regionserver.HStore(310): Store=51e4d6a6293430d21cdf75126c54f808/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:29,514 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:29,514 INFO [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] regionserver.HStore(310): Store=aaa51003a74d8671fa0840bbfcb549ac/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:29,514 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:29,515 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:29,516 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:29,516 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=127, resume processing ppid=126 2023-02-11 04:00:29,516 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=127, ppid=126, state=SUCCESS; OpenRegionProcedure d586dffdbc612899aa5c72d99bdc6519, server=jenkins-hbase4.apache.org,46851,1676088013694 in 169 msec 2023-02-11 04:00:29,517 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=128, resume processing ppid=122 2023-02-11 04:00:29,517 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=128, ppid=122, state=SUCCESS; OpenRegionProcedure 3452669615c81575c52ff96406a41e9e, server=jenkins-hbase4.apache.org,42385,1676088013771 in 171 msec 2023-02-11 04:00:29,518 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=126, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, ASSIGN in 335 msec 2023-02-11 04:00:29,518 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:29,518 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=130, resume processing ppid=124 2023-02-11 04:00:29,518 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=130, ppid=124, state=SUCCESS; OpenRegionProcedure e1d0ef015d92b032f67c5052a11865c4, server=jenkins-hbase4.apache.org,40829,1676088013739 in 170 msec 2023-02-11 04:00:29,519 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 51e4d6a6293430d21cdf75126c54f808; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70359317, jitterRate=0.048435524106025696}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:29,519 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 51e4d6a6293430d21cdf75126c54f808: Running coprocessor pre-open hook at 1676088029507Writing region info on filesystem at 1676088029507Initializing all the Stores at 1676088029508 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088029509 (+1 ms)Cleaning up temporary data from old regions at 1676088029515 (+6 ms)Cleaning up detritus from prior splits at 1676088029515Running coprocessor post-open hooks at 1676088029519 (+4 ms)Region opened successfully at 1676088029519 2023-02-11 04:00:29,519 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=122, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, ASSIGN in 336 msec 2023-02-11 04:00:29,520 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:29,520 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808., pid=131, masterSystemTime=1676088029492 2023-02-11 04:00:29,520 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=124, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, ASSIGN in 337 msec 2023-02-11 04:00:29,521 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened aaa51003a74d8671fa0840bbfcb549ac; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63105455, jitterRate=-0.05965544283390045}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:29,521 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for aaa51003a74d8671fa0840bbfcb549ac: Running coprocessor pre-open hook at 1676088029508Writing region info on filesystem at 1676088029508Initializing all the Stores at 1676088029509 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088029510 (+1 ms)Cleaning up temporary data from old regions at 1676088029517 (+7 ms)Cleaning up detritus from prior splits at 1676088029517Running coprocessor post-open hooks at 1676088029521 (+4 ms)Region opened successfully at 1676088029521 2023-02-11 04:00:29,521 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:29,521 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:29,521 DEBUG [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf 2023-02-11 04:00:29,522 DEBUG [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf 2023-02-11 04:00:29,522 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac., pid=129, masterSystemTime=1676088029493 2023-02-11 04:00:29,522 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=123 updating hbase:meta row=51e4d6a6293430d21cdf75126c54f808, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:29,522 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088029522"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088029522"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088029522"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088029522"}]},"ts":"1676088029522"} 2023-02-11 04:00:29,522 INFO [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 747a6fa324aa972cea6bbd307f3b1539 columnFamilyName cf 2023-02-11 04:00:29,523 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:29,523 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:29,523 INFO [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] regionserver.HStore(310): Store=747a6fa324aa972cea6bbd307f3b1539/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:29,523 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=125 updating hbase:meta row=aaa51003a74d8671fa0840bbfcb549ac, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:29,524 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088029523"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088029523"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088029523"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088029523"}]},"ts":"1676088029523"} 2023-02-11 04:00:29,524 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:29,526 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:29,526 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=131, resume processing ppid=123 2023-02-11 04:00:29,526 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=131, ppid=123, state=SUCCESS; OpenRegionProcedure 51e4d6a6293430d21cdf75126c54f808, server=jenkins-hbase4.apache.org,46851,1676088013694 in 180 msec 2023-02-11 04:00:29,527 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=129, resume processing ppid=125 2023-02-11 04:00:29,527 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=129, ppid=125, state=SUCCESS; OpenRegionProcedure aaa51003a74d8671fa0840bbfcb549ac, server=jenkins-hbase4.apache.org,42385,1676088013771 in 183 msec 2023-02-11 04:00:29,527 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=123, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, ASSIGN in 345 msec 2023-02-11 04:00:29,528 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=125, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, ASSIGN in 346 msec 2023-02-11 04:00:29,528 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:29,529 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 747a6fa324aa972cea6bbd307f3b1539; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62089918, jitterRate=-0.07478812336921692}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:29,529 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 747a6fa324aa972cea6bbd307f3b1539: Running coprocessor pre-open hook at 1676088029510Writing region info on filesystem at 1676088029510Initializing all the Stores at 1676088029510Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088029511 (+1 ms)Cleaning up temporary data from old regions at 1676088029526 (+15 ms)Cleaning up detritus from prior splits at 1676088029527 (+1 ms)Running coprocessor post-open hooks at 1676088029529 (+2 ms)Region opened successfully at 1676088029529 2023-02-11 04:00:29,530 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539., pid=132, masterSystemTime=1676088029495 2023-02-11 04:00:29,531 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:29,531 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:29,531 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=747a6fa324aa972cea6bbd307f3b1539, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:29,531 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088029531"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088029531"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088029531"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088029531"}]},"ts":"1676088029531"} 2023-02-11 04:00:29,533 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=132, resume processing ppid=121 2023-02-11 04:00:29,534 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=132, ppid=121, state=SUCCESS; OpenRegionProcedure 747a6fa324aa972cea6bbd307f3b1539, server=jenkins-hbase4.apache.org,40829,1676088013739 in 186 msec 2023-02-11 04:00:29,535 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=121, resume processing ppid=120 2023-02-11 04:00:29,535 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=121, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, ASSIGN in 353 msec 2023-02-11 04:00:29,535 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088029535"}]},"ts":"1676088029535"} 2023-02-11 04:00:29,537 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676088016466, state=ENABLED in hbase:meta 2023-02-11 04:00:29,538 INFO [PEWorker-3] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676088016466' was successfully enabled. 2023-02-11 04:00:29,540 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=120, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676088016466 in 365 msec 2023-02-11 04:00:29,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-11 04:00:29,928 INFO [Listener at localhost/33753] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676088016466, procId: 120 completed 2023-02-11 04:00:29,929 DEBUG [Listener at localhost/33753] zookeeper.ReadOnlyZKClient(139): Connect 0x43ec0260 to 127.0.0.1:63425 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 04:00:29,935 DEBUG [Listener at localhost/33753] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54f3b092, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 04:00:29,936 DEBUG [hconnection-0x58227bee-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 04:00:29,939 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47772, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 04:00:29,944 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 04:00:29,946 DEBUG [hconnection-0x58227bee-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 04:00:29,949 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35006, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 04:00:29,950 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 04:00:29,952 DEBUG [hconnection-0x58227bee-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 04:00:29,957 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37118, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 04:00:29,958 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 04:00:29,963 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 04:00:29,968 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 04:00:29,974 INFO [Listener at localhost/33753] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676088016466 2023-02-11 04:00:29,975 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676088016466 2023-02-11 04:00:29,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] procedure2.ProcedureExecutor(1029): Stored pid=133, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:29,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=133 2023-02-11 04:00:29,979 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088029979"}]},"ts":"1676088029979"} 2023-02-11 04:00:29,980 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676088016466, state=DISABLING in hbase:meta 2023-02-11 04:00:29,988 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676088016466 to state=DISABLING 2023-02-11 04:00:29,989 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=134, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, UNASSIGN}, {pid=135, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, UNASSIGN}, {pid=136, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, UNASSIGN}, {pid=137, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, UNASSIGN}, {pid=138, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, UNASSIGN}, {pid=139, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, UNASSIGN}] 2023-02-11 04:00:29,991 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=136, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, UNASSIGN 2023-02-11 04:00:29,991 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=137, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, UNASSIGN 2023-02-11 04:00:29,992 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=139, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, UNASSIGN 2023-02-11 04:00:29,992 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=138, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, UNASSIGN 2023-02-11 04:00:29,992 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=135, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, UNASSIGN 2023-02-11 04:00:29,992 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=51e4d6a6293430d21cdf75126c54f808, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:29,992 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=e1d0ef015d92b032f67c5052a11865c4, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:29,993 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088029992"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088029992"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088029992"}]},"ts":"1676088029992"} 2023-02-11 04:00:29,993 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088029992"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088029992"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088029992"}]},"ts":"1676088029992"} 2023-02-11 04:00:29,993 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=138 updating hbase:meta row=aaa51003a74d8671fa0840bbfcb549ac, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:29,993 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=139 updating hbase:meta row=d586dffdbc612899aa5c72d99bdc6519, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:29,993 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088029993"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088029993"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088029993"}]},"ts":"1676088029993"} 2023-02-11 04:00:29,993 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088029993"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088029993"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088029993"}]},"ts":"1676088029993"} 2023-02-11 04:00:29,993 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=135 updating hbase:meta row=3452669615c81575c52ff96406a41e9e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:29,994 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088029993"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088029993"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088029993"}]},"ts":"1676088029993"} 2023-02-11 04:00:29,994 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=140, ppid=136, state=RUNNABLE; CloseRegionProcedure 51e4d6a6293430d21cdf75126c54f808, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:29,995 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=141, ppid=137, state=RUNNABLE; CloseRegionProcedure e1d0ef015d92b032f67c5052a11865c4, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:29,996 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=142, ppid=138, state=RUNNABLE; CloseRegionProcedure aaa51003a74d8671fa0840bbfcb549ac, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:29,997 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=143, ppid=139, state=RUNNABLE; CloseRegionProcedure d586dffdbc612899aa5c72d99bdc6519, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:29,997 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=144, ppid=135, state=RUNNABLE; CloseRegionProcedure 3452669615c81575c52ff96406a41e9e, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:29,998 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=134, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, UNASSIGN 2023-02-11 04:00:29,999 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=134 updating hbase:meta row=747a6fa324aa972cea6bbd307f3b1539, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:29,999 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088029999"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088029999"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088029999"}]},"ts":"1676088029999"} 2023-02-11 04:00:30,000 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=145, ppid=134, state=RUNNABLE; CloseRegionProcedure 747a6fa324aa972cea6bbd307f3b1539, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:30,147 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:30,148 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 51e4d6a6293430d21cdf75126c54f808, disabling compactions & flushes 2023-02-11 04:00:30,148 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:30,148 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:30,148 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. after waiting 0 ms 2023-02-11 04:00:30,148 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:30,148 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:30,148 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:30,149 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 3452669615c81575c52ff96406a41e9e, disabling compactions & flushes 2023-02-11 04:00:30,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 747a6fa324aa972cea6bbd307f3b1539, disabling compactions & flushes 2023-02-11 04:00:30,150 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:30,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:30,150 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:30,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. after waiting 0 ms 2023-02-11 04:00:30,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:30,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:30,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. after waiting 0 ms 2023-02-11 04:00:30,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:30,153 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-11 04:00:30,153 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-11 04:00:30,153 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-11 04:00:30,154 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:30,154 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:30,154 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 51e4d6a6293430d21cdf75126c54f808: Waiting for close lock at 1676088030147Running coprocessor pre-close hooks at 1676088030147Disabling compacts and flushes for region at 1676088030147Disabling writes for close at 1676088030148 (+1 ms)Writing region close event to WAL at 1676088030148Running coprocessor post-close hooks at 1676088030154 (+6 ms)Closed at 1676088030154 2023-02-11 04:00:30,154 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:30,154 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 747a6fa324aa972cea6bbd307f3b1539: Waiting for close lock at 1676088030149Running coprocessor pre-close hooks at 1676088030149Disabling compacts and flushes for region at 1676088030149Disabling writes for close at 1676088030150 (+1 ms)Writing region close event to WAL at 1676088030151 (+1 ms)Running coprocessor post-close hooks at 1676088030154 (+3 ms)Closed at 1676088030154 2023-02-11 04:00:30,154 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 3452669615c81575c52ff96406a41e9e: Waiting for close lock at 1676088030148Running coprocessor pre-close hooks at 1676088030148Disabling compacts and flushes for region at 1676088030148Disabling writes for close at 1676088030150 (+2 ms)Writing region close event to WAL at 1676088030151 (+1 ms)Running coprocessor post-close hooks at 1676088030154 (+3 ms)Closed at 1676088030154 2023-02-11 04:00:30,155 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:30,155 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:30,156 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing d586dffdbc612899aa5c72d99bdc6519, disabling compactions & flushes 2023-02-11 04:00:30,156 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:30,156 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:30,156 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. after waiting 0 ms 2023-02-11 04:00:30,156 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:30,157 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=51e4d6a6293430d21cdf75126c54f808, regionState=CLOSED 2023-02-11 04:00:30,157 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088030156"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088030156"}]},"ts":"1676088030156"} 2023-02-11 04:00:30,157 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:30,158 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:30,158 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e1d0ef015d92b032f67c5052a11865c4, disabling compactions & flushes 2023-02-11 04:00:30,158 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:30,158 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:30,158 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. after waiting 0 ms 2023-02-11 04:00:30,158 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:30,158 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=134 updating hbase:meta row=747a6fa324aa972cea6bbd307f3b1539, regionState=CLOSED 2023-02-11 04:00:30,159 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088030158"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088030158"}]},"ts":"1676088030158"} 2023-02-11 04:00:30,159 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:30,159 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:30,160 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing aaa51003a74d8671fa0840bbfcb549ac, disabling compactions & flushes 2023-02-11 04:00:30,160 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:30,160 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:30,160 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. after waiting 0 ms 2023-02-11 04:00:30,160 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:30,160 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=135 updating hbase:meta row=3452669615c81575c52ff96406a41e9e, regionState=CLOSED 2023-02-11 04:00:30,160 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088030160"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088030160"}]},"ts":"1676088030160"} 2023-02-11 04:00:30,162 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=140, resume processing ppid=136 2023-02-11 04:00:30,163 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=140, ppid=136, state=SUCCESS; CloseRegionProcedure 51e4d6a6293430d21cdf75126c54f808, server=jenkins-hbase4.apache.org,46851,1676088013694 in 166 msec 2023-02-11 04:00:30,163 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=145, resume processing ppid=134 2023-02-11 04:00:30,163 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=145, ppid=134, state=SUCCESS; CloseRegionProcedure 747a6fa324aa972cea6bbd307f3b1539, server=jenkins-hbase4.apache.org,40829,1676088013739 in 161 msec 2023-02-11 04:00:30,164 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=144, resume processing ppid=135 2023-02-11 04:00:30,164 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=144, ppid=135, state=SUCCESS; CloseRegionProcedure 3452669615c81575c52ff96406a41e9e, server=jenkins-hbase4.apache.org,42385,1676088013771 in 165 msec 2023-02-11 04:00:30,164 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=136, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, UNASSIGN in 174 msec 2023-02-11 04:00:30,164 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=134, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, UNASSIGN in 174 msec 2023-02-11 04:00:30,165 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=135, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, UNASSIGN in 175 msec 2023-02-11 04:00:30,169 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-11 04:00:30,169 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-11 04:00:30,169 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-11 04:00:30,169 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:30,169 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:30,169 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for d586dffdbc612899aa5c72d99bdc6519: Waiting for close lock at 1676088030155Running coprocessor pre-close hooks at 1676088030155Disabling compacts and flushes for region at 1676088030155Disabling writes for close at 1676088030156 (+1 ms)Writing region close event to WAL at 1676088030157 (+1 ms)Running coprocessor post-close hooks at 1676088030169 (+12 ms)Closed at 1676088030169 2023-02-11 04:00:30,170 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for aaa51003a74d8671fa0840bbfcb549ac: Waiting for close lock at 1676088030159Running coprocessor pre-close hooks at 1676088030159Disabling compacts and flushes for region at 1676088030159Disabling writes for close at 1676088030160 (+1 ms)Writing region close event to WAL at 1676088030160Running coprocessor post-close hooks at 1676088030169 (+9 ms)Closed at 1676088030169 2023-02-11 04:00:30,169 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:30,170 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e1d0ef015d92b032f67c5052a11865c4: Waiting for close lock at 1676088030158Running coprocessor pre-close hooks at 1676088030158Disabling compacts and flushes for region at 1676088030158Disabling writes for close at 1676088030158Writing region close event to WAL at 1676088030160 (+2 ms)Running coprocessor post-close hooks at 1676088030169 (+9 ms)Closed at 1676088030169 2023-02-11 04:00:30,171 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:30,171 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=139 updating hbase:meta row=d586dffdbc612899aa5c72d99bdc6519, regionState=CLOSED 2023-02-11 04:00:30,171 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088030171"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088030171"}]},"ts":"1676088030171"} 2023-02-11 04:00:30,171 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:30,172 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=138 updating hbase:meta row=aaa51003a74d8671fa0840bbfcb549ac, regionState=CLOSED 2023-02-11 04:00:30,172 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088030172"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088030172"}]},"ts":"1676088030172"} 2023-02-11 04:00:30,172 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:30,172 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=e1d0ef015d92b032f67c5052a11865c4, regionState=CLOSED 2023-02-11 04:00:30,172 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088030172"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088030172"}]},"ts":"1676088030172"} 2023-02-11 04:00:30,174 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=143, resume processing ppid=139 2023-02-11 04:00:30,174 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=143, ppid=139, state=SUCCESS; CloseRegionProcedure d586dffdbc612899aa5c72d99bdc6519, server=jenkins-hbase4.apache.org,46851,1676088013694 in 175 msec 2023-02-11 04:00:30,174 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=142, resume processing ppid=138 2023-02-11 04:00:30,175 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=142, ppid=138, state=SUCCESS; CloseRegionProcedure aaa51003a74d8671fa0840bbfcb549ac, server=jenkins-hbase4.apache.org,42385,1676088013771 in 177 msec 2023-02-11 04:00:30,175 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=141, resume processing ppid=137 2023-02-11 04:00:30,175 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=139, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, UNASSIGN in 185 msec 2023-02-11 04:00:30,175 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=141, ppid=137, state=SUCCESS; CloseRegionProcedure e1d0ef015d92b032f67c5052a11865c4, server=jenkins-hbase4.apache.org,40829,1676088013739 in 179 msec 2023-02-11 04:00:30,176 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=138, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, UNASSIGN in 186 msec 2023-02-11 04:00:30,177 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=137, resume processing ppid=133 2023-02-11 04:00:30,177 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=137, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, UNASSIGN in 186 msec 2023-02-11 04:00:30,177 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088030177"}]},"ts":"1676088030177"} 2023-02-11 04:00:30,178 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676088016466, state=DISABLED in hbase:meta 2023-02-11 04:00:30,180 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676088016466 to state=DISABLED 2023-02-11 04:00:30,182 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=133, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676088016466 in 206 msec 2023-02-11 04:00:30,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=133 2023-02-11 04:00:30,230 INFO [Listener at localhost/33753] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676088016466, procId: 133 completed 2023-02-11 04:00:30,239 INFO [Listener at localhost/33753] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-snaptb1-1676088016466-1676088030239 2023-02-11 04:00:30,239 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=hbase-failsafe-snaptb1-1676088016466-1676088030239 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 } 2023-02-11 04:00:30,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676088030239 (current time:1676088030239). 2023-02-11 04:00:30,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-11 04:00:30,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] zookeeper.ReadOnlyZKClient(139): Connect 0x259f23b8 to 127.0.0.1:63425 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 04:00:30,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6d5daf99, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 04:00:30,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 04:00:30,247 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47786, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 04:00:30,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x259f23b8 to 127.0.0.1:63425 2023-02-11 04:00:30,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:00:30,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-11 04:00:30,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-snaptb1-1676088016466-1676088030239 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 } 2023-02-11 04:00:30,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] procedure2.ProcedureExecutor(1029): Stored pid=146, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676088016466, type=EXCLUSIVE 2023-02-11 04:00:30,251 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=146, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676088016466, type=EXCLUSIVE 2023-02-11 04:00:30,252 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=146, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676088016466, type=EXCLUSIVE; timeout=600000, timestamp=1676088630252 2023-02-11 04:00:30,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-snaptb1-1676088016466-1676088030239 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 } 2023-02-11 04:00:30,252 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-snaptb1-1676088016466-1676088030239 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676088016466 2023-02-11 04:00:30,252 DEBUG [Listener at localhost/33753] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-snaptb1-1676088016466-1676088030239 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-11 04:00:30,252 DEBUG [Listener at localhost/33753] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-11 04:00:30,261 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb1-1676088016466-1676088030239 table=testRestoreSnapshot-1676088016466 type=DISABLED ttl=0 } 2023-02-11 04:00:30,263 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:30,263 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:30,263 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:30,263 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:30,263 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:30,263 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:30,264 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:30,264 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:30,264 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:30,264 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:30,264 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:30,264 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:30,264 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:30,264 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:30,265 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 04:00:30,265 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 04:00:30,265 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 04:00:30,265 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 04:00:30,265 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 04:00:30,266 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 04:00:30,267 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 04:00:30,287 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-snaptb1-1676088016466-1676088030239 table=testRestoreSnapshot-1676088016466 type=DISABLED ttl=0 } as finished. 2023-02-11 04:00:30,287 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-snaptb1-1676088016466-1676088030239 2023-02-11 04:00:30,287 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676088016466-1676088030239 2023-02-11 04:00:30,304 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676088016466-1676088030239 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/hbase-failsafe-snaptb1-1676088016466-1676088030239 2023-02-11 04:00:30,352 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-11 04:00:30,502 DEBUG [Listener at localhost/33753] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 04:00:30,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb1-1676088016466-1676088030239 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 } is done 2023-02-11 04:00:30,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(404): Snapshoting '{ ss=hbase-failsafe-snaptb1-1676088016466-1676088030239 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 }' is still in progress! 2023-02-11 04:00:30,504 DEBUG [Listener at localhost/33753] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-11 04:00:30,724 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-snaptb1-1676088016466-1676088030239 of table testRestoreSnapshot-1676088016466 completed 2023-02-11 04:00:30,724 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676088016466-1676088030239 2023-02-11 04:00:30,724 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676088016466-1676088030239 2023-02-11 04:00:30,724 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-snaptb1-1676088016466-1676088030239 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676088016466 at 1676088030252Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb1-1676088016466-1676088030239 table=testRestoreSnapshot-1676088016466 type=DISABLED ttl=0 } at 1676088030261 (+9 ms)Consolidate snapshot: hbase-failsafe-snaptb1-1676088016466-1676088030239 at 1676088030287 (+26 ms)Loading Region manifests for hbase-failsafe-snaptb1-1676088016466-1676088030239 at 1676088030287Writing data manifest for hbase-failsafe-snaptb1-1676088016466-1676088030239 at 1676088030290 (+3 ms)Verifying snapshot: hbase-failsafe-snaptb1-1676088016466-1676088030239 at 1676088030299 (+9 ms)Snapshot hbase-failsafe-snaptb1-1676088016466-1676088030239 of table testRestoreSnapshot-1676088016466 completed at 1676088030724 (+425 ms) 2023-02-11 04:00:30,726 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=146, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676088016466, type=EXCLUSIVE 2023-02-11 04:00:30,728 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=146, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676088016466, type=EXCLUSIVE in 476 msec 2023-02-11 04:00:31,004 DEBUG [Listener at localhost/33753] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 04:00:31,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb1-1676088016466-1676088030239 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 } is done 2023-02-11 04:00:31,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-snaptb1-1676088016466-1676088030239 table=testRestoreSnapshot-1676088016466 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-11 04:00:31,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] procedure2.ProcedureExecutor(1029): Stored pid=147, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676088016466 snapshot=name: "snaptb1-1676088016466" table: "testRestoreSnapshot-1676088016466" creation_time: 1676088025361 type: DISABLED version: 2 ttl: 0 ) 2023-02-11 04:00:31,021 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(880): Restore snapshot=snaptb1-1676088016466 as table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:31,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=147 2023-02-11 04:00:31,037 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/.tabledesc/.tableinfo.0000000004 2023-02-11 04:00:31,038 DEBUG [PEWorker-1] util.FSTableDescriptors(519): Deleted hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/.tabledesc/.tableinfo.0000000003 2023-02-11 04:00:31,038 INFO [PEWorker-1] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/.tabledesc/.tableinfo.0000000004 2023-02-11 04:00:31,039 INFO [PEWorker-1] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=snaptb1-1676088016466 table=testRestoreSnapshot-1676088016466 type=DISABLED ttl=0 } 2023-02-11 04:00:31,041 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb1-1676088016466" table: "testRestoreSnapshot-1676088016466" creation_time: 1676088025361 type: DISABLED version: 2 ttl: 0 2023-02-11 04:00:31,041 DEBUG [PEWorker-1] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466 2023-02-11 04:00:31,048 DEBUG [PEWorker-1] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:31,049 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f of snapshot=snaptb1-1676088016466 to region=089c12f910ab0d90d02340003e54031f table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:31,054 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 of snapshot=snaptb1-1676088016466 to region=089c12f910ab0d90d02340003e54031f table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:31,058 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 of snapshot=snaptb1-1676088016466 to region=089c12f910ab0d90d02340003e54031f table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:31,061 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 of snapshot=snaptb1-1676088016466 to region=089c12f910ab0d90d02340003e54031f table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:31,064 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 of snapshot=snaptb1-1676088016466 to region=089c12f910ab0d90d02340003e54031f table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:31,067 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 of snapshot=snaptb1-1676088016466 to region=089c12f910ab0d90d02340003e54031f table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:31,071 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 of snapshot=snaptb1-1676088016466 to region=089c12f910ab0d90d02340003e54031f table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:31,074 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b of snapshot=snaptb1-1676088016466 to region=089c12f910ab0d90d02340003e54031f table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:31,077 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 of snapshot=snaptb1-1676088016466 to region=089c12f910ab0d90d02340003e54031f table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:31,082 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e of snapshot=snaptb1-1676088016466 to region=089c12f910ab0d90d02340003e54031f table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:31,086 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 of snapshot=snaptb1-1676088016466 to region=089c12f910ab0d90d02340003e54031f table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:31,089 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 of snapshot=snaptb1-1676088016466 to region=089c12f910ab0d90d02340003e54031f table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:31,092 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:31,093 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:31,093 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:31,093 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:31,093 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:31,093 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:31,096 DEBUG [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a11c828e49bd4e7a9cf05be9ff07ad7f of snapshot=snaptb1-1676088016466 to region=3452669615c81575c52ff96406a41e9e table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:31,097 DEBUG [RestoreSnapshot-pool-6] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 93d3b4e383ae4310a66075c83ee5c664 of snapshot=snaptb1-1676088016466 to region=e1d0ef015d92b032f67c5052a11865c4 table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:31,098 DEBUG [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 1b38064ae3304eca9c176d5669359823 of snapshot=snaptb1-1676088016466 to region=51e4d6a6293430d21cdf75126c54f808 table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:31,098 DEBUG [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d1e1bf338c724083a10375a4bd78b7d9 of snapshot=snaptb1-1676088016466 to region=d586dffdbc612899aa5c72d99bdc6519 table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:31,099 DEBUG [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 3f27b1f3105b45bf81c17ba9c704bf44 of snapshot=snaptb1-1676088016466 to region=aaa51003a74d8671fa0840bbfcb549ac table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:31,100 DEBUG [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 0a646c34d29c4963afc688a0da1fd6d4 of snapshot=snaptb1-1676088016466 to region=747a6fa324aa972cea6bbd307f3b1539 table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:31,113 DEBUG [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink f60cf166ab3e4854a8bb6a91f3df546f of snapshot=snaptb1-1676088016466 to region=3452669615c81575c52ff96406a41e9e table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:31,114 DEBUG [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 7dc891a823984ca99b9b428443d59643 of snapshot=snaptb1-1676088016466 to region=747a6fa324aa972cea6bbd307f3b1539 table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:31,115 DEBUG [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e6cd8a1c78d94d0c9da6aff3413bb91a of snapshot=snaptb1-1676088016466 to region=aaa51003a74d8671fa0840bbfcb549ac table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:31,115 DEBUG [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c1e521afb6ba4af19d964c416aa5a0de of snapshot=snaptb1-1676088016466 to region=51e4d6a6293430d21cdf75126c54f808 table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:31,126 DEBUG [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e928d1188fee412e9c504ba31bbb3ef6 of snapshot=snaptb1-1676088016466 to region=d586dffdbc612899aa5c72d99bdc6519 table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:31,126 DEBUG [RestoreSnapshot-pool-6] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e1f68e07f6354647bc060c949ab8bd60 of snapshot=snaptb1-1676088016466 to region=e1d0ef015d92b032f67c5052a11865c4 table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:31,131 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb1-1676088016466" table: "testRestoreSnapshot-1676088016466" creation_time: 1676088025361 type: DISABLED version: 2 ttl: 0 2023-02-11 04:00:31,132 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088031132"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:31,132 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088031132"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:31,132 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088031132"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:31,132 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088031132"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:31,132 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088031132"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:31,132 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088031132"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:31,135 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-11 04:00:31,135 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 3452669615c81575c52ff96406a41e9e, NAME => 'testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 51e4d6a6293430d21cdf75126c54f808, NAME => 'testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 747a6fa324aa972cea6bbd307f3b1539, NAME => 'testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => aaa51003a74d8671fa0840bbfcb549ac, NAME => 'testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => d586dffdbc612899aa5c72d99bdc6519, NAME => 'testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => e1d0ef015d92b032f67c5052a11865c4, NAME => 'testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.', STARTKEY => '3', ENDKEY => '4'}] 2023-02-11 04:00:31,135 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088031133"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088031133"}]},"ts":"1676088031133"} 2023-02-11 04:00:31,135 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088031133"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088031133"}]},"ts":"1676088031133"} 2023-02-11 04:00:31,135 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088031133"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088031133"}]},"ts":"1676088031133"} 2023-02-11 04:00:31,135 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088031133"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088031133"}]},"ts":"1676088031133"} 2023-02-11 04:00:31,135 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088031133"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088031133"}]},"ts":"1676088031133"} 2023-02-11 04:00:31,135 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088031133"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088031133"}]},"ts":"1676088031133"} 2023-02-11 04:00:31,137 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-11 04:00:31,137 INFO [PEWorker-1] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-11 04:00:31,137 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 3452669615c81575c52ff96406a41e9e, NAME => 'testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 51e4d6a6293430d21cdf75126c54f808, NAME => 'testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 747a6fa324aa972cea6bbd307f3b1539, NAME => 'testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => aaa51003a74d8671fa0840bbfcb549ac, NAME => 'testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => d586dffdbc612899aa5c72d99bdc6519, NAME => 'testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => e1d0ef015d92b032f67c5052a11865c4, NAME => 'testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.', STARTKEY => '3', ENDKEY => '4'}] 2023-02-11 04:00:31,137 INFO [PEWorker-1] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=snaptb1-1676088016466 table=testRestoreSnapshot-1676088016466 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676088016466 completed! 2023-02-11 04:00:31,139 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=147, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676088016466 snapshot=name: "snaptb1-1676088016466" table: "testRestoreSnapshot-1676088016466" creation_time: 1676088025361 type: DISABLED version: 2 ttl: 0 ) in 120 msec 2023-02-11 04:00:31,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=147 2023-02-11 04:00:31,274 INFO [Listener at localhost/33753] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676088016466, procId: 147 completed 2023-02-11 04:00:31,274 INFO [Listener at localhost/33753] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-snaptb1-1676088016466-1676088030239 2023-02-11 04:00:31,275 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "hbase-failsafe-snaptb1-1676088016466-1676088030239" 2023-02-11 04:00:31,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-snaptb1-1676088016466-1676088030239 2023-02-11 04:00:31,278 INFO [Listener at localhost/33753] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676088016466 2023-02-11 04:00:31,278 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676088016466 2023-02-11 04:00:31,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] procedure2.ProcedureExecutor(1029): Stored pid=148, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:31,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-11 04:00:31,281 INFO [PEWorker-5] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676088016466 2023-02-11 04:00:31,281 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088031281"}]},"ts":"1676088031281"} 2023-02-11 04:00:31,282 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676088016466, state=ENABLING in hbase:meta 2023-02-11 04:00:31,286 DEBUG [PEWorker-5] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-11 04:00:31,286 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, ASSIGN}, {pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, ASSIGN}, {pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, ASSIGN}, {pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, ASSIGN}, {pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, ASSIGN}, {pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, ASSIGN}] 2023-02-11 04:00:31,287 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, ASSIGN 2023-02-11 04:00:31,288 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, ASSIGN 2023-02-11 04:00:31,288 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, ASSIGN 2023-02-11 04:00:31,288 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, ASSIGN 2023-02-11 04:00:31,288 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, ASSIGN 2023-02-11 04:00:31,288 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 04:00:31,288 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 04:00:31,288 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 04:00:31,289 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 04:00:31,289 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 04:00:31,289 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, ASSIGN 2023-02-11 04:00:31,290 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 04:00:31,439 DEBUG [jenkins-hbase4:41713] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 04:00:31,439 DEBUG [jenkins-hbase4:41713] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 04:00:31,439 DEBUG [jenkins-hbase4:41713] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 04:00:31,439 DEBUG [jenkins-hbase4:41713] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 04:00:31,439 DEBUG [jenkins-hbase4:41713] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 04:00:31,442 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=150 updating hbase:meta row=3452669615c81575c52ff96406a41e9e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:31,442 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=aaa51003a74d8671fa0840bbfcb549ac, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:31,442 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088031442"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088031442"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088031442"}]},"ts":"1676088031442"} 2023-02-11 04:00:31,442 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=154 updating hbase:meta row=d586dffdbc612899aa5c72d99bdc6519, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:31,442 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=149 updating hbase:meta row=747a6fa324aa972cea6bbd307f3b1539, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:31,442 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=e1d0ef015d92b032f67c5052a11865c4, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:31,443 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088031442"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088031442"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088031442"}]},"ts":"1676088031442"} 2023-02-11 04:00:31,443 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088031442"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088031442"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088031442"}]},"ts":"1676088031442"} 2023-02-11 04:00:31,443 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088031442"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088031442"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088031442"}]},"ts":"1676088031442"} 2023-02-11 04:00:31,443 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088031442"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088031442"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088031442"}]},"ts":"1676088031442"} 2023-02-11 04:00:31,444 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=155, ppid=150, state=RUNNABLE; OpenRegionProcedure 3452669615c81575c52ff96406a41e9e, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:31,445 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=156, ppid=149, state=RUNNABLE; OpenRegionProcedure 747a6fa324aa972cea6bbd307f3b1539, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:31,446 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=157, ppid=152, state=RUNNABLE; OpenRegionProcedure e1d0ef015d92b032f67c5052a11865c4, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:31,446 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=158, ppid=154, state=RUNNABLE; OpenRegionProcedure d586dffdbc612899aa5c72d99bdc6519, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:31,447 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=159, ppid=153, state=RUNNABLE; OpenRegionProcedure aaa51003a74d8671fa0840bbfcb549ac, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:31,448 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=151 updating hbase:meta row=51e4d6a6293430d21cdf75126c54f808, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:31,448 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088031448"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088031448"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088031448"}]},"ts":"1676088031448"} 2023-02-11 04:00:31,449 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=160, ppid=151, state=RUNNABLE; OpenRegionProcedure 51e4d6a6293430d21cdf75126c54f808, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:31,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-11 04:00:31,598 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:31,598 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:31,599 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 3452669615c81575c52ff96406a41e9e, NAME => 'testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 04:00:31,599 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 747a6fa324aa972cea6bbd307f3b1539, NAME => 'testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 04:00:31,599 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:31,599 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:31,599 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:31,599 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:31,599 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:31,599 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:31,599 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:31,599 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:31,600 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:31,600 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d586dffdbc612899aa5c72d99bdc6519, NAME => 'testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 04:00:31,600 INFO [StoreOpener-3452669615c81575c52ff96406a41e9e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:31,600 INFO [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:31,600 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:31,601 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:31,601 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:31,601 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:31,601 DEBUG [StoreOpener-3452669615c81575c52ff96406a41e9e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf 2023-02-11 04:00:31,601 DEBUG [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf 2023-02-11 04:00:31,602 DEBUG [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf 2023-02-11 04:00:31,601 DEBUG [StoreOpener-3452669615c81575c52ff96406a41e9e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf 2023-02-11 04:00:31,602 INFO [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:31,602 INFO [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 747a6fa324aa972cea6bbd307f3b1539 columnFamilyName cf 2023-02-11 04:00:31,602 INFO [StoreOpener-3452669615c81575c52ff96406a41e9e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3452669615c81575c52ff96406a41e9e columnFamilyName cf 2023-02-11 04:00:31,602 DEBUG [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf 2023-02-11 04:00:31,603 DEBUG [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf 2023-02-11 04:00:31,603 INFO [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d586dffdbc612899aa5c72d99bdc6519 columnFamilyName cf 2023-02-11 04:00:31,620 DEBUG [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/testRestoreSnapshot-1676088016466=d586dffdbc612899aa5c72d99bdc6519-d1e1bf338c724083a10375a4bd78b7d9 2023-02-11 04:00:31,620 DEBUG [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/testRestoreSnapshot-1676088016466=747a6fa324aa972cea6bbd307f3b1539-0a646c34d29c4963afc688a0da1fd6d4 2023-02-11 04:00:31,622 DEBUG [StoreOpener-3452669615c81575c52ff96406a41e9e-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/testRestoreSnapshot-1676088016466=3452669615c81575c52ff96406a41e9e-a11c828e49bd4e7a9cf05be9ff07ad7f 2023-02-11 04:00:31,628 DEBUG [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/testRestoreSnapshot-1676088016466=d586dffdbc612899aa5c72d99bdc6519-e928d1188fee412e9c504ba31bbb3ef6 2023-02-11 04:00:31,628 DEBUG [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/testRestoreSnapshot-1676088016466=747a6fa324aa972cea6bbd307f3b1539-7dc891a823984ca99b9b428443d59643 2023-02-11 04:00:31,629 INFO [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] regionserver.HStore(310): Store=d586dffdbc612899aa5c72d99bdc6519/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:31,629 INFO [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] regionserver.HStore(310): Store=747a6fa324aa972cea6bbd307f3b1539/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:31,629 DEBUG [StoreOpener-3452669615c81575c52ff96406a41e9e-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/testRestoreSnapshot-1676088016466=3452669615c81575c52ff96406a41e9e-f60cf166ab3e4854a8bb6a91f3df546f 2023-02-11 04:00:31,629 INFO [StoreOpener-3452669615c81575c52ff96406a41e9e-1] regionserver.HStore(310): Store=3452669615c81575c52ff96406a41e9e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:31,630 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:31,630 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:31,630 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:31,631 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:31,631 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:31,631 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:31,634 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:31,634 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:31,634 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:31,635 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened d586dffdbc612899aa5c72d99bdc6519; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69207836, jitterRate=0.0312771201133728}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:31,635 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for d586dffdbc612899aa5c72d99bdc6519: Running coprocessor pre-open hook at 1676088031601Writing region info on filesystem at 1676088031601Initializing all the Stores at 1676088031601Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088031601Cleaning up temporary data from old regions at 1676088031631 (+30 ms)Cleaning up detritus from prior splits at 1676088031632 (+1 ms)Running coprocessor post-open hooks at 1676088031635 (+3 ms)Region opened successfully at 1676088031635 2023-02-11 04:00:31,635 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 3452669615c81575c52ff96406a41e9e; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74682110, jitterRate=0.11285015940666199}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:31,635 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 747a6fa324aa972cea6bbd307f3b1539; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72836854, jitterRate=0.08535370230674744}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:31,635 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 3452669615c81575c52ff96406a41e9e: Running coprocessor pre-open hook at 1676088031599Writing region info on filesystem at 1676088031599Initializing all the Stores at 1676088031600 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088031600Cleaning up temporary data from old regions at 1676088031632 (+32 ms)Cleaning up detritus from prior splits at 1676088031632Running coprocessor post-open hooks at 1676088031635 (+3 ms)Region opened successfully at 1676088031635 2023-02-11 04:00:31,635 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 747a6fa324aa972cea6bbd307f3b1539: Running coprocessor pre-open hook at 1676088031599Writing region info on filesystem at 1676088031599Initializing all the Stores at 1676088031600 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088031600Cleaning up temporary data from old regions at 1676088031632 (+32 ms)Cleaning up detritus from prior splits at 1676088031632Running coprocessor post-open hooks at 1676088031635 (+3 ms)Region opened successfully at 1676088031635 2023-02-11 04:00:31,636 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519., pid=158, masterSystemTime=1676088031598 2023-02-11 04:00:31,636 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e., pid=155, masterSystemTime=1676088031596 2023-02-11 04:00:31,636 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539., pid=156, masterSystemTime=1676088031596 2023-02-11 04:00:31,637 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:31,637 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:31,637 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:31,637 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 51e4d6a6293430d21cdf75126c54f808, NAME => 'testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 04:00:31,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:31,638 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=154 updating hbase:meta row=d586dffdbc612899aa5c72d99bdc6519, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:31,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:31,638 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088031638"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088031638"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088031638"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088031638"}]},"ts":"1676088031638"} 2023-02-11 04:00:31,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:31,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:31,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:31,639 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:31,639 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:31,639 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => aaa51003a74d8671fa0840bbfcb549ac, NAME => 'testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 04:00:31,639 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:31,639 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=150 updating hbase:meta row=3452669615c81575c52ff96406a41e9e, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:31,639 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:31,639 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088031639"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088031639"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088031639"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088031639"}]},"ts":"1676088031639"} 2023-02-11 04:00:31,639 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:31,639 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:31,640 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:31,640 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:31,640 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:31,640 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e1d0ef015d92b032f67c5052a11865c4, NAME => 'testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 04:00:31,640 INFO [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:31,640 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=149 updating hbase:meta row=747a6fa324aa972cea6bbd307f3b1539, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:31,640 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:31,640 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088031640"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088031640"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088031640"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088031640"}]},"ts":"1676088031640"} 2023-02-11 04:00:31,640 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:31,641 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:31,641 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:31,642 DEBUG [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf 2023-02-11 04:00:31,642 DEBUG [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf 2023-02-11 04:00:31,642 INFO [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 51e4d6a6293430d21cdf75126c54f808 columnFamilyName cf 2023-02-11 04:00:31,642 INFO [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:31,643 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=158, resume processing ppid=154 2023-02-11 04:00:31,643 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=158, ppid=154, state=SUCCESS; OpenRegionProcedure d586dffdbc612899aa5c72d99bdc6519, server=jenkins-hbase4.apache.org,40829,1676088013739 in 194 msec 2023-02-11 04:00:31,643 INFO [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:31,644 DEBUG [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf 2023-02-11 04:00:31,644 DEBUG [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf 2023-02-11 04:00:31,644 INFO [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e1d0ef015d92b032f67c5052a11865c4 columnFamilyName cf 2023-02-11 04:00:31,645 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=154, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, ASSIGN in 357 msec 2023-02-11 04:00:31,645 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=155, resume processing ppid=150 2023-02-11 04:00:31,645 DEBUG [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf 2023-02-11 04:00:31,645 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=155, ppid=150, state=SUCCESS; OpenRegionProcedure 3452669615c81575c52ff96406a41e9e, server=jenkins-hbase4.apache.org,46851,1676088013694 in 198 msec 2023-02-11 04:00:31,645 DEBUG [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf 2023-02-11 04:00:31,646 INFO [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region aaa51003a74d8671fa0840bbfcb549ac columnFamilyName cf 2023-02-11 04:00:31,646 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=156, resume processing ppid=149 2023-02-11 04:00:31,646 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=156, ppid=149, state=SUCCESS; OpenRegionProcedure 747a6fa324aa972cea6bbd307f3b1539, server=jenkins-hbase4.apache.org,42385,1676088013771 in 198 msec 2023-02-11 04:00:31,647 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=150, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, ASSIGN in 359 msec 2023-02-11 04:00:31,648 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=149, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, ASSIGN in 360 msec 2023-02-11 04:00:31,656 DEBUG [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/testRestoreSnapshot-1676088016466=51e4d6a6293430d21cdf75126c54f808-1b38064ae3304eca9c176d5669359823 2023-02-11 04:00:31,657 DEBUG [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/testRestoreSnapshot-1676088016466=e1d0ef015d92b032f67c5052a11865c4-93d3b4e383ae4310a66075c83ee5c664 2023-02-11 04:00:31,660 DEBUG [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/testRestoreSnapshot-1676088016466=aaa51003a74d8671fa0840bbfcb549ac-3f27b1f3105b45bf81c17ba9c704bf44 2023-02-11 04:00:31,668 DEBUG [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/testRestoreSnapshot-1676088016466=e1d0ef015d92b032f67c5052a11865c4-e1f68e07f6354647bc060c949ab8bd60 2023-02-11 04:00:31,668 INFO [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] regionserver.HStore(310): Store=e1d0ef015d92b032f67c5052a11865c4/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:31,669 DEBUG [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/testRestoreSnapshot-1676088016466=aaa51003a74d8671fa0840bbfcb549ac-e6cd8a1c78d94d0c9da6aff3413bb91a 2023-02-11 04:00:31,669 INFO [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] regionserver.HStore(310): Store=aaa51003a74d8671fa0840bbfcb549ac/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:31,669 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:31,669 DEBUG [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/testRestoreSnapshot-1676088016466=51e4d6a6293430d21cdf75126c54f808-c1e521afb6ba4af19d964c416aa5a0de 2023-02-11 04:00:31,670 INFO [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] regionserver.HStore(310): Store=51e4d6a6293430d21cdf75126c54f808/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:31,670 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:31,670 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:31,670 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:31,672 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:31,672 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:31,674 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:31,675 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:31,675 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:31,675 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e1d0ef015d92b032f67c5052a11865c4; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66610037, jitterRate=-0.007433101534843445}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:31,675 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e1d0ef015d92b032f67c5052a11865c4: Running coprocessor pre-open hook at 1676088031641Writing region info on filesystem at 1676088031641Initializing all the Stores at 1676088031641Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088031642 (+1 ms)Cleaning up temporary data from old regions at 1676088031671 (+29 ms)Cleaning up detritus from prior splits at 1676088031672 (+1 ms)Running coprocessor post-open hooks at 1676088031675 (+3 ms)Region opened successfully at 1676088031675 2023-02-11 04:00:31,676 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 51e4d6a6293430d21cdf75126c54f808; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64592726, jitterRate=-0.037493377923965454}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:31,676 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 51e4d6a6293430d21cdf75126c54f808: Running coprocessor pre-open hook at 1676088031638Writing region info on filesystem at 1676088031638Initializing all the Stores at 1676088031639 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088031639Cleaning up temporary data from old regions at 1676088031672 (+33 ms)Cleaning up detritus from prior splits at 1676088031673 (+1 ms)Running coprocessor post-open hooks at 1676088031676 (+3 ms)Region opened successfully at 1676088031676 2023-02-11 04:00:31,676 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4., pid=157, masterSystemTime=1676088031596 2023-02-11 04:00:31,677 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808., pid=160, masterSystemTime=1676088031598 2023-02-11 04:00:31,677 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened aaa51003a74d8671fa0840bbfcb549ac; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63767821, jitterRate=-0.049785420298576355}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:31,677 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for aaa51003a74d8671fa0840bbfcb549ac: Running coprocessor pre-open hook at 1676088031640Writing region info on filesystem at 1676088031640Initializing all the Stores at 1676088031641 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088031641Cleaning up temporary data from old regions at 1676088031672 (+31 ms)Cleaning up detritus from prior splits at 1676088031673 (+1 ms)Running coprocessor post-open hooks at 1676088031677 (+4 ms)Region opened successfully at 1676088031677 2023-02-11 04:00:31,678 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac., pid=159, masterSystemTime=1676088031596 2023-02-11 04:00:31,678 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:31,678 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:31,678 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=e1d0ef015d92b032f67c5052a11865c4, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:31,678 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:31,679 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088031678"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088031678"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088031678"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088031678"}]},"ts":"1676088031678"} 2023-02-11 04:00:31,679 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:31,679 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=151 updating hbase:meta row=51e4d6a6293430d21cdf75126c54f808, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:31,679 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088031679"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088031679"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088031679"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088031679"}]},"ts":"1676088031679"} 2023-02-11 04:00:31,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:31,680 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:31,680 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=aaa51003a74d8671fa0840bbfcb549ac, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:31,680 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088031680"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088031680"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088031680"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088031680"}]},"ts":"1676088031680"} 2023-02-11 04:00:31,683 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=157, resume processing ppid=152 2023-02-11 04:00:31,683 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=157, ppid=152, state=SUCCESS; OpenRegionProcedure e1d0ef015d92b032f67c5052a11865c4, server=jenkins-hbase4.apache.org,42385,1676088013771 in 234 msec 2023-02-11 04:00:31,683 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=160, resume processing ppid=151 2023-02-11 04:00:31,684 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=160, ppid=151, state=SUCCESS; OpenRegionProcedure 51e4d6a6293430d21cdf75126c54f808, server=jenkins-hbase4.apache.org,40829,1676088013739 in 232 msec 2023-02-11 04:00:31,684 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=152, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, ASSIGN in 397 msec 2023-02-11 04:00:31,684 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=159, resume processing ppid=153 2023-02-11 04:00:31,685 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=159, ppid=153, state=SUCCESS; OpenRegionProcedure aaa51003a74d8671fa0840bbfcb549ac, server=jenkins-hbase4.apache.org,46851,1676088013694 in 235 msec 2023-02-11 04:00:31,685 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=151, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, ASSIGN in 397 msec 2023-02-11 04:00:31,686 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=153, resume processing ppid=148 2023-02-11 04:00:31,686 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=153, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, ASSIGN in 399 msec 2023-02-11 04:00:31,687 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088031687"}]},"ts":"1676088031687"} 2023-02-11 04:00:31,688 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676088016466, state=ENABLED in hbase:meta 2023-02-11 04:00:31,690 INFO [PEWorker-5] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676088016466' was successfully enabled. 2023-02-11 04:00:31,692 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=148, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676088016466 in 412 msec 2023-02-11 04:00:32,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-11 04:00:32,033 INFO [Listener at localhost/33753] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676088016466, procId: 148 completed 2023-02-11 04:00:32,033 DEBUG [Listener at localhost/33753] zookeeper.ReadOnlyZKClient(139): Connect 0x70b95fe7 to 127.0.0.1:63425 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 04:00:32,039 DEBUG [Listener at localhost/33753] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ad94238, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 04:00:32,040 DEBUG [hconnection-0x3a5b8d75-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 04:00:32,043 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47794, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 04:00:32,046 DEBUG [hconnection-0x3a5b8d75-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 04:00:32,049 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35008, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 04:00:32,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,155 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x251148a9 to 127.0.0.1:63425 2023-02-11 04:00:32,156 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:00:32,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,157 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x43ec0260 to 127.0.0.1:63425 2023-02-11 04:00:32,157 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:00:32,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,254 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 04:00:32,257 DEBUG [hconnection-0x3a5b8d75-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 04:00:32,260 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37126, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 04:00:32,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,456 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 04:00:32,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,681 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 04:00:32,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,890 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 04:00:32,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:32,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,042 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 04:00:33,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:33,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:34,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:35,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:35,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:35,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:35,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:35,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:35,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:35,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:35,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:35,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:35,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:35,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:35,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:35,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:35,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:35,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:35,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:35,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:35,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:35,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:35,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:35,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:35,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:35,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:35,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:35,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:35,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:35,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:35,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:35,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:35,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:35,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:35,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:35,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:35,098 INFO [Listener at localhost/33753] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,098 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] procedure2.ProcedureExecutor(1029): Stored pid=161, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=161 2023-02-11 04:00:35,102 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088035102"}]},"ts":"1676088035102"} 2023-02-11 04:00:35,103 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676088016466, state=DISABLING in hbase:meta 2023-02-11 04:00:35,105 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676088016466 to state=DISABLING 2023-02-11 04:00:35,106 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=162, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, UNASSIGN}, {pid=163, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, UNASSIGN}, {pid=164, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, UNASSIGN}, {pid=165, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, UNASSIGN}, {pid=166, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, UNASSIGN}, {pid=167, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, UNASSIGN}] 2023-02-11 04:00:35,108 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=165, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, UNASSIGN 2023-02-11 04:00:35,108 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=163, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, UNASSIGN 2023-02-11 04:00:35,108 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=162, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, UNASSIGN 2023-02-11 04:00:35,108 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=164, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, UNASSIGN 2023-02-11 04:00:35,108 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=167, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, UNASSIGN 2023-02-11 04:00:35,108 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=165 updating hbase:meta row=e1d0ef015d92b032f67c5052a11865c4, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:35,109 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088035108"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088035108"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088035108"}]},"ts":"1676088035108"} 2023-02-11 04:00:35,109 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=163 updating hbase:meta row=3452669615c81575c52ff96406a41e9e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:35,109 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088035109"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088035109"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088035109"}]},"ts":"1676088035109"} 2023-02-11 04:00:35,109 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=162 updating hbase:meta row=747a6fa324aa972cea6bbd307f3b1539, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:35,109 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088035109"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088035109"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088035109"}]},"ts":"1676088035109"} 2023-02-11 04:00:35,109 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=164 updating hbase:meta row=51e4d6a6293430d21cdf75126c54f808, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:35,109 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=167 updating hbase:meta row=d586dffdbc612899aa5c72d99bdc6519, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:35,110 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088035109"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088035109"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088035109"}]},"ts":"1676088035109"} 2023-02-11 04:00:35,110 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088035109"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088035109"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088035109"}]},"ts":"1676088035109"} 2023-02-11 04:00:35,110 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=168, ppid=165, state=RUNNABLE; CloseRegionProcedure e1d0ef015d92b032f67c5052a11865c4, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:35,111 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=169, ppid=163, state=RUNNABLE; CloseRegionProcedure 3452669615c81575c52ff96406a41e9e, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:35,112 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=170, ppid=162, state=RUNNABLE; CloseRegionProcedure 747a6fa324aa972cea6bbd307f3b1539, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:35,113 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=171, ppid=164, state=RUNNABLE; CloseRegionProcedure 51e4d6a6293430d21cdf75126c54f808, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:35,114 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=172, ppid=167, state=RUNNABLE; CloseRegionProcedure d586dffdbc612899aa5c72d99bdc6519, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:35,115 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=166, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, UNASSIGN 2023-02-11 04:00:35,116 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=166 updating hbase:meta row=aaa51003a74d8671fa0840bbfcb549ac, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:35,116 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088035116"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088035116"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088035116"}]},"ts":"1676088035116"} 2023-02-11 04:00:35,118 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=173, ppid=166, state=RUNNABLE; CloseRegionProcedure aaa51003a74d8671fa0840bbfcb549ac, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:35,264 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:35,265 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e1d0ef015d92b032f67c5052a11865c4, disabling compactions & flushes 2023-02-11 04:00:35,265 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:35,265 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:35,265 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. after waiting 0 ms 2023-02-11 04:00:35,265 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:35,265 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:35,265 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:35,267 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing d586dffdbc612899aa5c72d99bdc6519, disabling compactions & flushes 2023-02-11 04:00:35,267 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 3452669615c81575c52ff96406a41e9e, disabling compactions & flushes 2023-02-11 04:00:35,267 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:35,267 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:35,267 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:35,267 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. after waiting 0 ms 2023-02-11 04:00:35,267 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:35,267 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:35,267 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. after waiting 0 ms 2023-02-11 04:00:35,267 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:35,282 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-11 04:00:35,282 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-11 04:00:35,282 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:35,282 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:35,282 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for d586dffdbc612899aa5c72d99bdc6519: Waiting for close lock at 1676088035265Running coprocessor pre-close hooks at 1676088035265Disabling compacts and flushes for region at 1676088035265Disabling writes for close at 1676088035267 (+2 ms)Writing region close event to WAL at 1676088035276 (+9 ms)Running coprocessor post-close hooks at 1676088035282 (+6 ms)Closed at 1676088035282 2023-02-11 04:00:35,283 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e1d0ef015d92b032f67c5052a11865c4: Waiting for close lock at 1676088035264Running coprocessor pre-close hooks at 1676088035264Disabling compacts and flushes for region at 1676088035264Disabling writes for close at 1676088035265 (+1 ms)Writing region close event to WAL at 1676088035276 (+11 ms)Running coprocessor post-close hooks at 1676088035282 (+6 ms)Closed at 1676088035282 2023-02-11 04:00:35,285 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:35,285 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:35,285 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=165 updating hbase:meta row=e1d0ef015d92b032f67c5052a11865c4, regionState=CLOSED 2023-02-11 04:00:35,286 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 747a6fa324aa972cea6bbd307f3b1539, disabling compactions & flushes 2023-02-11 04:00:35,286 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:35,286 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:35,286 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. after waiting 0 ms 2023-02-11 04:00:35,286 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:35,286 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:35,286 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088035285"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088035285"}]},"ts":"1676088035285"} 2023-02-11 04:00:35,286 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:35,286 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=167 updating hbase:meta row=d586dffdbc612899aa5c72d99bdc6519, regionState=CLOSED 2023-02-11 04:00:35,287 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088035286"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088035286"}]},"ts":"1676088035286"} 2023-02-11 04:00:35,288 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 51e4d6a6293430d21cdf75126c54f808, disabling compactions & flushes 2023-02-11 04:00:35,288 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:35,288 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:35,288 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. after waiting 0 ms 2023-02-11 04:00:35,288 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:35,289 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-11 04:00:35,290 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:35,290 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 3452669615c81575c52ff96406a41e9e: Waiting for close lock at 1676088035265Running coprocessor pre-close hooks at 1676088035265Disabling compacts and flushes for region at 1676088035265Disabling writes for close at 1676088035267 (+2 ms)Writing region close event to WAL at 1676088035276 (+9 ms)Running coprocessor post-close hooks at 1676088035290 (+14 ms)Closed at 1676088035290 2023-02-11 04:00:35,297 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-11 04:00:35,298 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:35,298 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 51e4d6a6293430d21cdf75126c54f808: Waiting for close lock at 1676088035286Running coprocessor pre-close hooks at 1676088035286Disabling compacts and flushes for region at 1676088035286Disabling writes for close at 1676088035288 (+2 ms)Writing region close event to WAL at 1676088035294 (+6 ms)Running coprocessor post-close hooks at 1676088035298 (+4 ms)Closed at 1676088035298 2023-02-11 04:00:35,300 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=168, resume processing ppid=165 2023-02-11 04:00:35,300 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=168, ppid=165, state=SUCCESS; CloseRegionProcedure e1d0ef015d92b032f67c5052a11865c4, server=jenkins-hbase4.apache.org,42385,1676088013771 in 179 msec 2023-02-11 04:00:35,301 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:35,301 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:35,302 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing aaa51003a74d8671fa0840bbfcb549ac, disabling compactions & flushes 2023-02-11 04:00:35,302 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:35,302 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:35,302 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. after waiting 0 ms 2023-02-11 04:00:35,302 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:35,302 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=163 updating hbase:meta row=3452669615c81575c52ff96406a41e9e, regionState=CLOSED 2023-02-11 04:00:35,302 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088035302"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088035302"}]},"ts":"1676088035302"} 2023-02-11 04:00:35,303 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:35,303 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=172, resume processing ppid=167 2023-02-11 04:00:35,303 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=172, ppid=167, state=SUCCESS; CloseRegionProcedure d586dffdbc612899aa5c72d99bdc6519, server=jenkins-hbase4.apache.org,40829,1676088013739 in 181 msec 2023-02-11 04:00:35,303 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=164 updating hbase:meta row=51e4d6a6293430d21cdf75126c54f808, regionState=CLOSED 2023-02-11 04:00:35,303 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=165, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, UNASSIGN in 195 msec 2023-02-11 04:00:35,303 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088035303"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088035303"}]},"ts":"1676088035303"} 2023-02-11 04:00:35,313 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=167, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, UNASSIGN in 197 msec 2023-02-11 04:00:35,314 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-11 04:00:35,315 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:35,315 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=169, resume processing ppid=163 2023-02-11 04:00:35,315 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 747a6fa324aa972cea6bbd307f3b1539: Waiting for close lock at 1676088035285Running coprocessor pre-close hooks at 1676088035285Disabling compacts and flushes for region at 1676088035285Disabling writes for close at 1676088035286 (+1 ms)Writing region close event to WAL at 1676088035294 (+8 ms)Running coprocessor post-close hooks at 1676088035315 (+21 ms)Closed at 1676088035315 2023-02-11 04:00:35,315 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=169, ppid=163, state=SUCCESS; CloseRegionProcedure 3452669615c81575c52ff96406a41e9e, server=jenkins-hbase4.apache.org,46851,1676088013694 in 193 msec 2023-02-11 04:00:35,315 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=171, resume processing ppid=164 2023-02-11 04:00:35,316 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=171, ppid=164, state=SUCCESS; CloseRegionProcedure 51e4d6a6293430d21cdf75126c54f808, server=jenkins-hbase4.apache.org,40829,1676088013739 in 192 msec 2023-02-11 04:00:35,317 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:35,317 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=164, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, UNASSIGN in 209 msec 2023-02-11 04:00:35,317 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=163, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, UNASSIGN in 209 msec 2023-02-11 04:00:35,317 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=162 updating hbase:meta row=747a6fa324aa972cea6bbd307f3b1539, regionState=CLOSED 2023-02-11 04:00:35,317 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088035317"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088035317"}]},"ts":"1676088035317"} 2023-02-11 04:00:35,320 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-11 04:00:35,321 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:35,321 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for aaa51003a74d8671fa0840bbfcb549ac: Waiting for close lock at 1676088035301Running coprocessor pre-close hooks at 1676088035301Disabling compacts and flushes for region at 1676088035301Disabling writes for close at 1676088035302 (+1 ms)Writing region close event to WAL at 1676088035315 (+13 ms)Running coprocessor post-close hooks at 1676088035321 (+6 ms)Closed at 1676088035321 2023-02-11 04:00:35,322 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=170, resume processing ppid=162 2023-02-11 04:00:35,322 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=170, ppid=162, state=SUCCESS; CloseRegionProcedure 747a6fa324aa972cea6bbd307f3b1539, server=jenkins-hbase4.apache.org,42385,1676088013771 in 207 msec 2023-02-11 04:00:35,323 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:35,323 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=162, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, UNASSIGN in 216 msec 2023-02-11 04:00:35,323 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=166 updating hbase:meta row=aaa51003a74d8671fa0840bbfcb549ac, regionState=CLOSED 2023-02-11 04:00:35,323 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088035323"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088035323"}]},"ts":"1676088035323"} 2023-02-11 04:00:35,326 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=173, resume processing ppid=166 2023-02-11 04:00:35,326 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=173, ppid=166, state=SUCCESS; CloseRegionProcedure aaa51003a74d8671fa0840bbfcb549ac, server=jenkins-hbase4.apache.org,46851,1676088013694 in 207 msec 2023-02-11 04:00:35,327 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=166, resume processing ppid=161 2023-02-11 04:00:35,327 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=166, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, UNASSIGN in 220 msec 2023-02-11 04:00:35,328 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088035328"}]},"ts":"1676088035328"} 2023-02-11 04:00:35,329 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676088016466, state=DISABLED in hbase:meta 2023-02-11 04:00:35,331 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676088016466 to state=DISABLED 2023-02-11 04:00:35,333 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=161, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676088016466 in 234 msec 2023-02-11 04:00:35,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=161 2023-02-11 04:00:35,353 INFO [Listener at localhost/33753] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676088016466, procId: 161 completed 2023-02-11 04:00:35,357 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.HMaster$5(2215): Client=jenkins//172.31.14.131 delete testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] procedure2.ProcedureExecutor(1029): Stored pid=174, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,365 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=174, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,367 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=174, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=174 2023-02-11 04:00:35,374 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:35,374 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:35,374 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:35,374 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:35,374 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:35,374 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:35,376 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf, FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/recovered.edits] 2023-02-11 04:00:35,376 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf, FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/recovered.edits] 2023-02-11 04:00:35,376 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf, FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/recovered.edits] 2023-02-11 04:00:35,376 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf, FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/recovered.edits] 2023-02-11 04:00:35,376 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf, FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/recovered.edits] 2023-02-11 04:00:35,376 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf, FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/recovered.edits] 2023-02-11 04:00:35,380 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/testRestoreSnapshot-1676088016466=aaa51003a74d8671fa0840bbfcb549ac-3f27b1f3105b45bf81c17ba9c704bf44 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/testRestoreSnapshot-1676088016466=aaa51003a74d8671fa0840bbfcb549ac-3f27b1f3105b45bf81c17ba9c704bf44 2023-02-11 04:00:35,380 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/testRestoreSnapshot-1676088016466=51e4d6a6293430d21cdf75126c54f808-1b38064ae3304eca9c176d5669359823 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/testRestoreSnapshot-1676088016466=51e4d6a6293430d21cdf75126c54f808-1b38064ae3304eca9c176d5669359823 2023-02-11 04:00:35,380 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/testRestoreSnapshot-1676088016466=d586dffdbc612899aa5c72d99bdc6519-d1e1bf338c724083a10375a4bd78b7d9 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/testRestoreSnapshot-1676088016466=d586dffdbc612899aa5c72d99bdc6519-d1e1bf338c724083a10375a4bd78b7d9 2023-02-11 04:00:35,380 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/testRestoreSnapshot-1676088016466=3452669615c81575c52ff96406a41e9e-a11c828e49bd4e7a9cf05be9ff07ad7f to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/testRestoreSnapshot-1676088016466=3452669615c81575c52ff96406a41e9e-a11c828e49bd4e7a9cf05be9ff07ad7f 2023-02-11 04:00:35,380 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/testRestoreSnapshot-1676088016466=747a6fa324aa972cea6bbd307f3b1539-0a646c34d29c4963afc688a0da1fd6d4 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/testRestoreSnapshot-1676088016466=747a6fa324aa972cea6bbd307f3b1539-0a646c34d29c4963afc688a0da1fd6d4 2023-02-11 04:00:35,381 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/testRestoreSnapshot-1676088016466=e1d0ef015d92b032f67c5052a11865c4-93d3b4e383ae4310a66075c83ee5c664 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/testRestoreSnapshot-1676088016466=e1d0ef015d92b032f67c5052a11865c4-93d3b4e383ae4310a66075c83ee5c664 2023-02-11 04:00:35,382 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/testRestoreSnapshot-1676088016466=3452669615c81575c52ff96406a41e9e-f60cf166ab3e4854a8bb6a91f3df546f to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/testRestoreSnapshot-1676088016466=3452669615c81575c52ff96406a41e9e-f60cf166ab3e4854a8bb6a91f3df546f 2023-02-11 04:00:35,382 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/testRestoreSnapshot-1676088016466=aaa51003a74d8671fa0840bbfcb549ac-e6cd8a1c78d94d0c9da6aff3413bb91a to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/testRestoreSnapshot-1676088016466=aaa51003a74d8671fa0840bbfcb549ac-e6cd8a1c78d94d0c9da6aff3413bb91a 2023-02-11 04:00:35,382 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/testRestoreSnapshot-1676088016466=d586dffdbc612899aa5c72d99bdc6519-e928d1188fee412e9c504ba31bbb3ef6 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/testRestoreSnapshot-1676088016466=d586dffdbc612899aa5c72d99bdc6519-e928d1188fee412e9c504ba31bbb3ef6 2023-02-11 04:00:35,382 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/testRestoreSnapshot-1676088016466=51e4d6a6293430d21cdf75126c54f808-c1e521afb6ba4af19d964c416aa5a0de to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/testRestoreSnapshot-1676088016466=51e4d6a6293430d21cdf75126c54f808-c1e521afb6ba4af19d964c416aa5a0de 2023-02-11 04:00:35,382 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/testRestoreSnapshot-1676088016466=e1d0ef015d92b032f67c5052a11865c4-e1f68e07f6354647bc060c949ab8bd60 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/testRestoreSnapshot-1676088016466=e1d0ef015d92b032f67c5052a11865c4-e1f68e07f6354647bc060c949ab8bd60 2023-02-11 04:00:35,382 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/testRestoreSnapshot-1676088016466=747a6fa324aa972cea6bbd307f3b1539-7dc891a823984ca99b9b428443d59643 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/testRestoreSnapshot-1676088016466=747a6fa324aa972cea6bbd307f3b1539-7dc891a823984ca99b9b428443d59643 2023-02-11 04:00:35,387 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/recovered.edits/27.seqid to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/recovered.edits/27.seqid 2023-02-11 04:00:35,387 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/recovered.edits/27.seqid to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/recovered.edits/27.seqid 2023-02-11 04:00:35,387 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/recovered.edits/27.seqid to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/recovered.edits/27.seqid 2023-02-11 04:00:35,387 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/recovered.edits/27.seqid to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/recovered.edits/27.seqid 2023-02-11 04:00:35,387 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/recovered.edits/27.seqid to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/recovered.edits/27.seqid 2023-02-11 04:00:35,387 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/recovered.edits/27.seqid to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/recovered.edits/27.seqid 2023-02-11 04:00:35,387 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:35,388 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:35,388 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:35,388 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:35,388 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:35,388 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:35,388 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676088016466 regions 2023-02-11 04:00:35,389 DEBUG [PEWorker-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f 2023-02-11 04:00:35,389 DEBUG [PEWorker-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf] 2023-02-11 04:00:35,392 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f 2023-02-11 04:00:35,393 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 2023-02-11 04:00:35,394 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 2023-02-11 04:00:35,395 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 2023-02-11 04:00:35,396 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 2023-02-11 04:00:35,397 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 2023-02-11 04:00:35,398 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 2023-02-11 04:00:35,399 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b 2023-02-11 04:00:35,400 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 2023-02-11 04:00:35,401 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e 2023-02-11 04:00:35,402 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 2023-02-11 04:00:35,403 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 2023-02-11 04:00:35,404 DEBUG [PEWorker-3] backup.HFileArchiver(596): Deleted hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f 2023-02-11 04:00:35,405 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=174, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,427 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testRestoreSnapshot-1676088016466 from hbase:meta 2023-02-11 04:00:35,429 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'testRestoreSnapshot-1676088016466' descriptor. 2023-02-11 04:00:35,430 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=174, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,430 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'testRestoreSnapshot-1676088016466' from region states. 2023-02-11 04:00:35,430 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088035430"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:35,431 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088035430"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:35,431 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088035430"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:35,431 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088035430"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:35,431 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088035430"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:35,431 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088035430"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:35,432 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-11 04:00:35,432 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 747a6fa324aa972cea6bbd307f3b1539, NAME => 'testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 3452669615c81575c52ff96406a41e9e, NAME => 'testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 51e4d6a6293430d21cdf75126c54f808, NAME => 'testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => e1d0ef015d92b032f67c5052a11865c4, NAME => 'testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => aaa51003a74d8671fa0840bbfcb549ac, NAME => 'testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => d586dffdbc612899aa5c72d99bdc6519, NAME => 'testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.', STARTKEY => '5', ENDKEY => ''}] 2023-02-11 04:00:35,432 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'testRestoreSnapshot-1676088016466' as deleted. 2023-02-11 04:00:35,433 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676088035432"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:35,434 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table testRestoreSnapshot-1676088016466 state from META 2023-02-11 04:00:35,436 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=174, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,436 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=174, state=SUCCESS; DeleteTableProcedure table=testRestoreSnapshot-1676088016466 in 78 msec 2023-02-11 04:00:35,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=174 2023-02-11 04:00:35,619 INFO [Listener at localhost/33753] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRestoreSnapshot-1676088016466, procId: 174 completed 2023-02-11 04:00:35,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] procedure2.ProcedureExecutor(1029): Stored pid=175, state=RUNNABLE:CLONE_SNAPSHOT_PRE_OPERATION; CloneSnapshotProcedure (table=testRestoreSnapshot-1676088016466 snapshot=name: "snaptb1-1676088016466" table: "testRestoreSnapshot-1676088016466" creation_time: 1676088025361 type: DISABLED version: 2 ttl: 0 ) 2023-02-11 04:00:35,646 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(750): Clone snapshot=snaptb1-1676088016466 as table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-11 04:00:35,657 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-11 04:00:35,659 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/.tabledesc/.tableinfo.0000000001 2023-02-11 04:00:35,661 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb1-1676088016466" table: "testRestoreSnapshot-1676088016466" creation_time: 1676088025361 type: DISABLED version: 2 ttl: 0 2023-02-11 04:00:35,661 DEBUG [PEWorker-4] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,661 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f from cloned region in snapshot snaptb1-1676088016466 to table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,664 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 from cloned region in snapshot snaptb1-1676088016466 to table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,666 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 from cloned region in snapshot snaptb1-1676088016466 to table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,668 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 from cloned region in snapshot snaptb1-1676088016466 to table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,670 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 from cloned region in snapshot snaptb1-1676088016466 to table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,673 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 from cloned region in snapshot snaptb1-1676088016466 to table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,675 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 from cloned region in snapshot snaptb1-1676088016466 to table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,677 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b from cloned region in snapshot snaptb1-1676088016466 to table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,679 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 from cloned region in snapshot snaptb1-1676088016466 to table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,682 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e from cloned region in snapshot snaptb1-1676088016466 to table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,685 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 from cloned region in snapshot snaptb1-1676088016466 to table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,687 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 from cloned region in snapshot snaptb1-1676088016466 to table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,690 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(239): region to add: aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:35,690 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(239): region to add: d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:35,690 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(239): region to add: 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:35,690 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(239): region to add: 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:35,690 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(239): region to add: 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:35,690 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(239): region to add: e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:35,690 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(585): clone region=aaa51003a74d8671fa0840bbfcb549ac as aaa51003a74d8671fa0840bbfcb549ac in snapshot snaptb1-1676088016466 2023-02-11 04:00:35,690 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(585): clone region=d586dffdbc612899aa5c72d99bdc6519 as d586dffdbc612899aa5c72d99bdc6519 in snapshot snaptb1-1676088016466 2023-02-11 04:00:35,690 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(585): clone region=51e4d6a6293430d21cdf75126c54f808 as 51e4d6a6293430d21cdf75126c54f808 in snapshot snaptb1-1676088016466 2023-02-11 04:00:35,690 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(585): clone region=3452669615c81575c52ff96406a41e9e as 3452669615c81575c52ff96406a41e9e in snapshot snaptb1-1676088016466 2023-02-11 04:00:35,690 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(585): clone region=747a6fa324aa972cea6bbd307f3b1539 as 747a6fa324aa972cea6bbd307f3b1539 in snapshot snaptb1-1676088016466 2023-02-11 04:00:35,690 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(585): clone region=e1d0ef015d92b032f67c5052a11865c4 as e1d0ef015d92b032f67c5052a11865c4 in snapshot snaptb1-1676088016466 2023-02-11 04:00:35,692 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(7671): creating {ENCODED => aaa51003a74d8671fa0840bbfcb549ac, NAME => 'testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testRestoreSnapshot-1676088016466', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp 2023-02-11 04:00:35,692 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(7671): creating {ENCODED => 51e4d6a6293430d21cdf75126c54f808, NAME => 'testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testRestoreSnapshot-1676088016466', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp 2023-02-11 04:00:35,693 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(7671): creating {ENCODED => d586dffdbc612899aa5c72d99bdc6519, NAME => 'testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testRestoreSnapshot-1676088016466', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp 2023-02-11 04:00:35,697 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(7671): creating {ENCODED => 747a6fa324aa972cea6bbd307f3b1539, NAME => 'testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testRestoreSnapshot-1676088016466', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp 2023-02-11 04:00:35,697 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(7671): creating {ENCODED => 3452669615c81575c52ff96406a41e9e, NAME => 'testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testRestoreSnapshot-1676088016466', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp 2023-02-11 04:00:35,698 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(7671): creating {ENCODED => e1d0ef015d92b032f67c5052a11865c4, NAME => 'testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testRestoreSnapshot-1676088016466', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp 2023-02-11 04:00:35,733 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:35,733 INFO [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d1e1bf338c724083a10375a4bd78b7d9 from cloned region in snapshot snaptb1-1676088016466 to table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,734 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:35,734 INFO [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 1b38064ae3304eca9c176d5669359823 from cloned region in snapshot snaptb1-1676088016466 to table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,738 INFO [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c1e521afb6ba4af19d964c416aa5a0de from cloned region in snapshot snaptb1-1676088016466 to table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,739 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:35,739 INFO [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 93d3b4e383ae4310a66075c83ee5c664 from cloned region in snapshot snaptb1-1676088016466 to table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,739 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:35,739 INFO [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a11c828e49bd4e7a9cf05be9ff07ad7f from cloned region in snapshot snaptb1-1676088016466 to table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,744 INFO [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e1f68e07f6354647bc060c949ab8bd60 from cloned region in snapshot snaptb1-1676088016466 to table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,744 INFO [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink f60cf166ab3e4854a8bb6a91f3df546f from cloned region in snapshot snaptb1-1676088016466 to table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,744 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1603): Closing 51e4d6a6293430d21cdf75126c54f808, disabling compactions & flushes 2023-02-11 04:00:35,744 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:35,744 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:35,744 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. after waiting 0 ms 2023-02-11 04:00:35,744 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:35,744 INFO [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e928d1188fee412e9c504ba31bbb3ef6 from cloned region in snapshot snaptb1-1676088016466 to table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,744 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:35,744 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1557): Region close journal for 51e4d6a6293430d21cdf75126c54f808: Waiting for close lock at 1676088035744Disabling compacts and flushes for region at 1676088035744Disabling writes for close at 1676088035744Writing region close event to WAL at 1676088035744Closed at 1676088035744 2023-02-11 04:00:35,745 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:35,746 INFO [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 0a646c34d29c4963afc688a0da1fd6d4 from cloned region in snapshot snaptb1-1676088016466 to table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,746 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:35,746 INFO [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 3f27b1f3105b45bf81c17ba9c704bf44 from cloned region in snapshot snaptb1-1676088016466 to table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,749 INFO [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e6cd8a1c78d94d0c9da6aff3413bb91a from cloned region in snapshot snaptb1-1676088016466 to table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,749 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1603): Closing 3452669615c81575c52ff96406a41e9e, disabling compactions & flushes 2023-02-11 04:00:35,749 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1603): Closing e1d0ef015d92b032f67c5052a11865c4, disabling compactions & flushes 2023-02-11 04:00:35,749 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:35,749 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:35,749 INFO [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 7dc891a823984ca99b9b428443d59643 from cloned region in snapshot snaptb1-1676088016466 to table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:35,750 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1603): Closing d586dffdbc612899aa5c72d99bdc6519, disabling compactions & flushes 2023-02-11 04:00:35,749 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:35,750 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:35,749 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:35,750 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:35,750 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. after waiting 0 ms 2023-02-11 04:00:35,750 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. after waiting 0 ms 2023-02-11 04:00:35,750 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. after waiting 0 ms 2023-02-11 04:00:35,750 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:35,750 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:35,750 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:35,750 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:35,750 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1557): Region close journal for d586dffdbc612899aa5c72d99bdc6519: Waiting for close lock at 1676088035750Disabling compacts and flushes for region at 1676088035750Disabling writes for close at 1676088035750Writing region close event to WAL at 1676088035750Closed at 1676088035750 2023-02-11 04:00:35,750 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:35,750 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:35,750 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1557): Region close journal for 3452669615c81575c52ff96406a41e9e: Waiting for close lock at 1676088035749Disabling compacts and flushes for region at 1676088035749Disabling writes for close at 1676088035750 (+1 ms)Writing region close event to WAL at 1676088035750Closed at 1676088035750 2023-02-11 04:00:35,750 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1557): Region close journal for e1d0ef015d92b032f67c5052a11865c4: Waiting for close lock at 1676088035749Disabling compacts and flushes for region at 1676088035749Disabling writes for close at 1676088035750 (+1 ms)Writing region close event to WAL at 1676088035750Closed at 1676088035750 2023-02-11 04:00:35,752 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1603): Closing aaa51003a74d8671fa0840bbfcb549ac, disabling compactions & flushes 2023-02-11 04:00:35,752 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1603): Closing 747a6fa324aa972cea6bbd307f3b1539, disabling compactions & flushes 2023-02-11 04:00:35,752 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:35,752 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:35,752 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:35,752 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:35,752 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. after waiting 0 ms 2023-02-11 04:00:35,752 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:35,752 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. after waiting 0 ms 2023-02-11 04:00:35,752 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:35,752 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:35,752 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1557): Region close journal for aaa51003a74d8671fa0840bbfcb549ac: Waiting for close lock at 1676088035752Disabling compacts and flushes for region at 1676088035752Disabling writes for close at 1676088035752Writing region close event to WAL at 1676088035752Closed at 1676088035752 2023-02-11 04:00:35,752 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:35,752 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1557): Region close journal for 747a6fa324aa972cea6bbd307f3b1539: Waiting for close lock at 1676088035752Disabling compacts and flushes for region at 1676088035752Disabling writes for close at 1676088035752Writing region close event to WAL at 1676088035752Closed at 1676088035752 2023-02-11 04:00:35,752 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb1-1676088016466" table: "testRestoreSnapshot-1676088016466" creation_time: 1676088025361 type: DISABLED version: 2 ttl: 0 2023-02-11 04:00:35,752 INFO [PEWorker-4] procedure.CloneSnapshotProcedure$1(411): Clone snapshot=snaptb1-1676088016466 on table=testRestoreSnapshot-1676088016466 completed! 2023-02-11 04:00:35,756 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088035756"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088035756"}]},"ts":"1676088035756"} 2023-02-11 04:00:35,757 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088035756"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088035756"}]},"ts":"1676088035756"} 2023-02-11 04:00:35,757 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088035756"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088035756"}]},"ts":"1676088035756"} 2023-02-11 04:00:35,757 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088035756"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088035756"}]},"ts":"1676088035756"} 2023-02-11 04:00:35,757 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088035756"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088035756"}]},"ts":"1676088035756"} 2023-02-11 04:00:35,757 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088035756"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088035756"}]},"ts":"1676088035756"} 2023-02-11 04:00:35,759 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-11 04:00:35,760 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088035760"}]},"ts":"1676088035760"} 2023-02-11 04:00:35,761 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676088016466, state=ENABLING in hbase:meta 2023-02-11 04:00:35,766 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 04:00:35,766 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 04:00:35,766 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 04:00:35,766 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 04:00:35,766 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 04:00:35,767 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, ASSIGN}, {pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, ASSIGN}, {pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, ASSIGN}, {pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, ASSIGN}, {pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, ASSIGN}, {pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, ASSIGN}] 2023-02-11 04:00:35,769 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, ASSIGN 2023-02-11 04:00:35,769 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, ASSIGN 2023-02-11 04:00:35,769 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, ASSIGN 2023-02-11 04:00:35,769 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, ASSIGN 2023-02-11 04:00:35,770 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40829,1676088013739; forceNewPlan=false, retain=false 2023-02-11 04:00:35,770 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46851,1676088013694; forceNewPlan=false, retain=false 2023-02-11 04:00:35,770 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42385,1676088013771; forceNewPlan=false, retain=false 2023-02-11 04:00:35,770 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46851,1676088013694; forceNewPlan=false, retain=false 2023-02-11 04:00:35,770 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, ASSIGN 2023-02-11 04:00:35,771 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, ASSIGN 2023-02-11 04:00:35,771 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42385,1676088013771; forceNewPlan=false, retain=false 2023-02-11 04:00:35,771 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40829,1676088013739; forceNewPlan=false, retain=false 2023-02-11 04:00:35,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-11 04:00:35,920 INFO [jenkins-hbase4:41713] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-11 04:00:35,923 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=179 updating hbase:meta row=e1d0ef015d92b032f67c5052a11865c4, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:35,923 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=176 updating hbase:meta row=747a6fa324aa972cea6bbd307f3b1539, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:35,923 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088035923"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088035923"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088035923"}]},"ts":"1676088035923"} 2023-02-11 04:00:35,923 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=178 updating hbase:meta row=51e4d6a6293430d21cdf75126c54f808, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:35,923 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=181 updating hbase:meta row=d586dffdbc612899aa5c72d99bdc6519, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:35,923 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=177 updating hbase:meta row=3452669615c81575c52ff96406a41e9e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:35,924 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088035923"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088035923"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088035923"}]},"ts":"1676088035923"} 2023-02-11 04:00:35,924 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088035923"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088035923"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088035923"}]},"ts":"1676088035923"} 2023-02-11 04:00:35,923 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088035923"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088035923"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088035923"}]},"ts":"1676088035923"} 2023-02-11 04:00:35,923 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088035923"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088035923"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088035923"}]},"ts":"1676088035923"} 2023-02-11 04:00:35,925 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=182, ppid=179, state=RUNNABLE; OpenRegionProcedure e1d0ef015d92b032f67c5052a11865c4, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:35,928 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=183, ppid=181, state=RUNNABLE; OpenRegionProcedure d586dffdbc612899aa5c72d99bdc6519, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:35,928 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=184, ppid=178, state=RUNNABLE; OpenRegionProcedure 51e4d6a6293430d21cdf75126c54f808, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:35,929 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=185, ppid=177, state=RUNNABLE; OpenRegionProcedure 3452669615c81575c52ff96406a41e9e, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:35,930 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=186, ppid=176, state=RUNNABLE; OpenRegionProcedure 747a6fa324aa972cea6bbd307f3b1539, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:35,930 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=180 updating hbase:meta row=aaa51003a74d8671fa0840bbfcb549ac, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:35,931 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088035930"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088035930"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088035930"}]},"ts":"1676088035930"} 2023-02-11 04:00:35,932 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=187, ppid=180, state=RUNNABLE; OpenRegionProcedure aaa51003a74d8671fa0840bbfcb549ac, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:36,080 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:36,080 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 51e4d6a6293430d21cdf75126c54f808, NAME => 'testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 04:00:36,081 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:36,081 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 747a6fa324aa972cea6bbd307f3b1539, NAME => 'testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 04:00:36,081 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:36,081 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:36,081 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:36,081 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:36,081 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:36,081 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:36,081 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:36,081 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:36,082 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:36,082 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 3452669615c81575c52ff96406a41e9e, NAME => 'testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 04:00:36,082 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:36,082 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:36,082 INFO [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:36,082 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:36,082 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:36,082 INFO [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:36,083 DEBUG [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf 2023-02-11 04:00:36,083 DEBUG [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf 2023-02-11 04:00:36,083 INFO [StoreOpener-3452669615c81575c52ff96406a41e9e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:36,083 INFO [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 51e4d6a6293430d21cdf75126c54f808 columnFamilyName cf 2023-02-11 04:00:36,084 DEBUG [StoreOpener-3452669615c81575c52ff96406a41e9e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf 2023-02-11 04:00:36,084 DEBUG [StoreOpener-3452669615c81575c52ff96406a41e9e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf 2023-02-11 04:00:36,085 INFO [StoreOpener-3452669615c81575c52ff96406a41e9e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3452669615c81575c52ff96406a41e9e columnFamilyName cf 2023-02-11 04:00:36,085 DEBUG [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf 2023-02-11 04:00:36,085 DEBUG [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf 2023-02-11 04:00:36,085 INFO [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 747a6fa324aa972cea6bbd307f3b1539 columnFamilyName cf 2023-02-11 04:00:36,101 DEBUG [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/testRestoreSnapshot-1676088016466=51e4d6a6293430d21cdf75126c54f808-1b38064ae3304eca9c176d5669359823 2023-02-11 04:00:36,104 DEBUG [StoreOpener-3452669615c81575c52ff96406a41e9e-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/testRestoreSnapshot-1676088016466=3452669615c81575c52ff96406a41e9e-a11c828e49bd4e7a9cf05be9ff07ad7f 2023-02-11 04:00:36,105 DEBUG [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/testRestoreSnapshot-1676088016466=747a6fa324aa972cea6bbd307f3b1539-0a646c34d29c4963afc688a0da1fd6d4 2023-02-11 04:00:36,109 DEBUG [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/testRestoreSnapshot-1676088016466=51e4d6a6293430d21cdf75126c54f808-c1e521afb6ba4af19d964c416aa5a0de 2023-02-11 04:00:36,109 INFO [StoreOpener-51e4d6a6293430d21cdf75126c54f808-1] regionserver.HStore(310): Store=51e4d6a6293430d21cdf75126c54f808/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:36,110 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:36,111 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:36,112 DEBUG [StoreOpener-3452669615c81575c52ff96406a41e9e-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/testRestoreSnapshot-1676088016466=3452669615c81575c52ff96406a41e9e-f60cf166ab3e4854a8bb6a91f3df546f 2023-02-11 04:00:36,112 INFO [StoreOpener-3452669615c81575c52ff96406a41e9e-1] regionserver.HStore(310): Store=3452669615c81575c52ff96406a41e9e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:36,113 DEBUG [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/testRestoreSnapshot-1676088016466=747a6fa324aa972cea6bbd307f3b1539-7dc891a823984ca99b9b428443d59643 2023-02-11 04:00:36,113 INFO [StoreOpener-747a6fa324aa972cea6bbd307f3b1539-1] regionserver.HStore(310): Store=747a6fa324aa972cea6bbd307f3b1539/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:36,113 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:36,114 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:36,114 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:36,114 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:36,114 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:36,116 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:36,116 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-11 04:00:36,116 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:36,117 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 51e4d6a6293430d21cdf75126c54f808; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72611793, jitterRate=0.08200003206729889}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:36,117 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 51e4d6a6293430d21cdf75126c54f808: Running coprocessor pre-open hook at 1676088036081Writing region info on filesystem at 1676088036081Initializing all the Stores at 1676088036082 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088036082Cleaning up temporary data from old regions at 1676088036111 (+29 ms)Cleaning up detritus from prior splits at 1676088036112 (+1 ms)Running coprocessor post-open hooks at 1676088036117 (+5 ms)Region opened successfully at 1676088036117 2023-02-11 04:00:36,117 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808., pid=184, masterSystemTime=1676088036078 2023-02-11 04:00:36,118 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-11 04:00:36,118 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-11 04:00:36,118 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 3452669615c81575c52ff96406a41e9e; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58861564, jitterRate=-0.12289434671401978}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:36,119 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 3452669615c81575c52ff96406a41e9e: Running coprocessor pre-open hook at 1676088036082Writing region info on filesystem at 1676088036082Initializing all the Stores at 1676088036083 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088036083Cleaning up temporary data from old regions at 1676088036114 (+31 ms)Cleaning up detritus from prior splits at 1676088036114Running coprocessor post-open hooks at 1676088036119 (+5 ms)Region opened successfully at 1676088036119 2023-02-11 04:00:36,119 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:36,119 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:36,119 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:36,119 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e1d0ef015d92b032f67c5052a11865c4, NAME => 'testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 04:00:36,119 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e., pid=185, masterSystemTime=1676088036080 2023-02-11 04:00:36,120 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:36,120 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=178 updating hbase:meta row=51e4d6a6293430d21cdf75126c54f808, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:36,120 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088036120"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088036120"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088036120"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088036120"}]},"ts":"1676088036120"} 2023-02-11 04:00:36,120 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:36,120 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:36,120 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:36,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:36,121 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:36,121 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:36,121 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 747a6fa324aa972cea6bbd307f3b1539; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73263770, jitterRate=0.0917152464389801}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:36,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => aaa51003a74d8671fa0840bbfcb549ac, NAME => 'testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 04:00:36,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 747a6fa324aa972cea6bbd307f3b1539: Running coprocessor pre-open hook at 1676088036081Writing region info on filesystem at 1676088036081Initializing all the Stores at 1676088036082 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088036082Cleaning up temporary data from old regions at 1676088036114 (+32 ms)Cleaning up detritus from prior splits at 1676088036115 (+1 ms)Running coprocessor post-open hooks at 1676088036121 (+6 ms)Region opened successfully at 1676088036121 2023-02-11 04:00:36,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:36,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:36,122 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:36,122 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:36,122 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=177 updating hbase:meta row=3452669615c81575c52ff96406a41e9e, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:36,122 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539., pid=186, masterSystemTime=1676088036079 2023-02-11 04:00:36,122 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088036122"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088036122"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088036122"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088036122"}]},"ts":"1676088036122"} 2023-02-11 04:00:36,122 INFO [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:36,123 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=184, resume processing ppid=178 2023-02-11 04:00:36,123 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:36,123 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=184, ppid=178, state=SUCCESS; OpenRegionProcedure 51e4d6a6293430d21cdf75126c54f808, server=jenkins-hbase4.apache.org,46851,1676088013694 in 193 msec 2023-02-11 04:00:36,123 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:36,123 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:36,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d586dffdbc612899aa5c72d99bdc6519, NAME => 'testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 04:00:36,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676088016466 d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:36,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:36,124 INFO [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:36,123 DEBUG [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf 2023-02-11 04:00:36,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:36,124 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=176 updating hbase:meta row=747a6fa324aa972cea6bbd307f3b1539, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:36,124 DEBUG [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf 2023-02-11 04:00:36,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:36,125 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088036124"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088036124"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088036124"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088036124"}]},"ts":"1676088036124"} 2023-02-11 04:00:36,125 DEBUG [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf 2023-02-11 04:00:36,125 DEBUG [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf 2023-02-11 04:00:36,125 INFO [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e1d0ef015d92b032f67c5052a11865c4 columnFamilyName cf 2023-02-11 04:00:36,125 INFO [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region aaa51003a74d8671fa0840bbfcb549ac columnFamilyName cf 2023-02-11 04:00:36,126 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=178, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, ASSIGN in 356 msec 2023-02-11 04:00:36,126 INFO [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:36,126 DEBUG [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf 2023-02-11 04:00:36,127 DEBUG [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf 2023-02-11 04:00:36,127 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=185, resume processing ppid=177 2023-02-11 04:00:36,127 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=185, ppid=177, state=SUCCESS; OpenRegionProcedure 3452669615c81575c52ff96406a41e9e, server=jenkins-hbase4.apache.org,40829,1676088013739 in 194 msec 2023-02-11 04:00:36,127 INFO [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d586dffdbc612899aa5c72d99bdc6519 columnFamilyName cf 2023-02-11 04:00:36,129 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=177, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, ASSIGN in 360 msec 2023-02-11 04:00:36,130 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=186, resume processing ppid=176 2023-02-11 04:00:36,130 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=186, ppid=176, state=SUCCESS; OpenRegionProcedure 747a6fa324aa972cea6bbd307f3b1539, server=jenkins-hbase4.apache.org,42385,1676088013771 in 197 msec 2023-02-11 04:00:36,131 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=176, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, ASSIGN in 363 msec 2023-02-11 04:00:36,140 DEBUG [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/testRestoreSnapshot-1676088016466=e1d0ef015d92b032f67c5052a11865c4-93d3b4e383ae4310a66075c83ee5c664 2023-02-11 04:00:36,141 DEBUG [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/testRestoreSnapshot-1676088016466=d586dffdbc612899aa5c72d99bdc6519-d1e1bf338c724083a10375a4bd78b7d9 2023-02-11 04:00:36,144 DEBUG [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/testRestoreSnapshot-1676088016466=aaa51003a74d8671fa0840bbfcb549ac-3f27b1f3105b45bf81c17ba9c704bf44 2023-02-11 04:00:36,146 DEBUG [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/testRestoreSnapshot-1676088016466=e1d0ef015d92b032f67c5052a11865c4-e1f68e07f6354647bc060c949ab8bd60 2023-02-11 04:00:36,146 INFO [StoreOpener-e1d0ef015d92b032f67c5052a11865c4-1] regionserver.HStore(310): Store=e1d0ef015d92b032f67c5052a11865c4/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:36,151 DEBUG [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/testRestoreSnapshot-1676088016466=d586dffdbc612899aa5c72d99bdc6519-e928d1188fee412e9c504ba31bbb3ef6 2023-02-11 04:00:36,151 INFO [StoreOpener-d586dffdbc612899aa5c72d99bdc6519-1] regionserver.HStore(310): Store=d586dffdbc612899aa5c72d99bdc6519/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:36,151 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:36,151 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:36,152 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:36,153 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:36,156 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:36,157 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:36,159 DEBUG [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/testRestoreSnapshot-1676088016466=aaa51003a74d8671fa0840bbfcb549ac-e6cd8a1c78d94d0c9da6aff3413bb91a 2023-02-11 04:00:36,164 INFO [StoreOpener-aaa51003a74d8671fa0840bbfcb549ac-1] regionserver.HStore(310): Store=aaa51003a74d8671fa0840bbfcb549ac/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:36,164 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-11 04:00:36,165 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e1d0ef015d92b032f67c5052a11865c4; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72430364, jitterRate=0.0792965292930603}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:36,165 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e1d0ef015d92b032f67c5052a11865c4: Running coprocessor pre-open hook at 1676088036120Writing region info on filesystem at 1676088036120Initializing all the Stores at 1676088036122 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088036122Cleaning up temporary data from old regions at 1676088036152 (+30 ms)Cleaning up detritus from prior splits at 1676088036153 (+1 ms)Running coprocessor post-open hooks at 1676088036165 (+12 ms)Region opened successfully at 1676088036165 2023-02-11 04:00:36,165 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-11 04:00:36,166 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:36,166 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4., pid=182, masterSystemTime=1676088036078 2023-02-11 04:00:36,167 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened d586dffdbc612899aa5c72d99bdc6519; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72868052, jitterRate=0.0858185887336731}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:36,167 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for d586dffdbc612899aa5c72d99bdc6519: Running coprocessor pre-open hook at 1676088036125Writing region info on filesystem at 1676088036125Initializing all the Stores at 1676088036125Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088036125Cleaning up temporary data from old regions at 1676088036154 (+29 ms)Cleaning up detritus from prior splits at 1676088036154Running coprocessor post-open hooks at 1676088036167 (+13 ms)Region opened successfully at 1676088036167 2023-02-11 04:00:36,167 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:36,168 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519., pid=183, masterSystemTime=1676088036079 2023-02-11 04:00:36,170 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=179 updating hbase:meta row=e1d0ef015d92b032f67c5052a11865c4, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:36,170 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:36,170 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088036169"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088036169"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088036169"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088036169"}]},"ts":"1676088036169"} 2023-02-11 04:00:36,170 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:36,170 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:36,170 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:36,171 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=181 updating hbase:meta row=d586dffdbc612899aa5c72d99bdc6519, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:36,171 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088036171"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088036171"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088036171"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088036171"}]},"ts":"1676088036171"} 2023-02-11 04:00:36,171 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:36,173 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=182, resume processing ppid=179 2023-02-11 04:00:36,174 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=182, ppid=179, state=SUCCESS; OpenRegionProcedure e1d0ef015d92b032f67c5052a11865c4, server=jenkins-hbase4.apache.org,46851,1676088013694 in 246 msec 2023-02-11 04:00:36,174 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-11 04:00:36,175 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened aaa51003a74d8671fa0840bbfcb549ac; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70387783, jitterRate=0.04885970056056976}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:36,175 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for aaa51003a74d8671fa0840bbfcb549ac: Running coprocessor pre-open hook at 1676088036122Writing region info on filesystem at 1676088036122Initializing all the Stores at 1676088036124 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088036124Cleaning up temporary data from old regions at 1676088036168 (+44 ms)Cleaning up detritus from prior splits at 1676088036169 (+1 ms)Running coprocessor post-open hooks at 1676088036175 (+6 ms)Region opened successfully at 1676088036175 2023-02-11 04:00:36,176 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=183, resume processing ppid=181 2023-02-11 04:00:36,176 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=183, ppid=181, state=SUCCESS; OpenRegionProcedure d586dffdbc612899aa5c72d99bdc6519, server=jenkins-hbase4.apache.org,42385,1676088013771 in 247 msec 2023-02-11 04:00:36,177 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac., pid=187, masterSystemTime=1676088036080 2023-02-11 04:00:36,178 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=179, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, ASSIGN in 407 msec 2023-02-11 04:00:36,179 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:36,179 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:36,179 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=181, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, ASSIGN in 409 msec 2023-02-11 04:00:36,179 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=180 updating hbase:meta row=aaa51003a74d8671fa0840bbfcb549ac, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:36,180 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088036179"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088036179"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088036179"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088036179"}]},"ts":"1676088036179"} 2023-02-11 04:00:36,182 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=187, resume processing ppid=180 2023-02-11 04:00:36,182 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=187, ppid=180, state=SUCCESS; OpenRegionProcedure aaa51003a74d8671fa0840bbfcb549ac, server=jenkins-hbase4.apache.org,40829,1676088013739 in 249 msec 2023-02-11 04:00:36,184 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=180, resume processing ppid=175 2023-02-11 04:00:36,184 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=180, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, ASSIGN in 415 msec 2023-02-11 04:00:36,185 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088036185"}]},"ts":"1676088036185"} 2023-02-11 04:00:36,186 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676088016466, state=ENABLED in hbase:meta 2023-02-11 04:00:36,195 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=175, state=SUCCESS; CloneSnapshotProcedure (table=testRestoreSnapshot-1676088016466 snapshot=name: "snaptb1-1676088016466" table: "testRestoreSnapshot-1676088016466" creation_time: 1676088025361 type: DISABLED version: 2 ttl: 0 ) in 553 msec 2023-02-11 04:00:36,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-11 04:00:36,400 INFO [Listener at localhost/33753] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676088016466, procId: 175 completed 2023-02-11 04:00:36,401 DEBUG [Listener at localhost/33753] zookeeper.ReadOnlyZKClient(139): Connect 0x69705b98 to 127.0.0.1:63425 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 04:00:36,409 DEBUG [Listener at localhost/33753] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3e1138f2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 04:00:36,411 DEBUG [hconnection-0x594888d9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 04:00:36,414 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39090, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 04:00:36,417 DEBUG [hconnection-0x594888d9-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 04:00:36,420 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44480, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 04:00:36,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,596 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 04:00:36,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,792 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 04:00:36,794 DEBUG [hconnection-0x594888d9-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 04:00:36,795 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40388, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 04:00:36,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:36,994 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 04:00:37,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,173 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 04:00:37,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,302 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 04:00:37,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,814 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70b95fe7 to 127.0.0.1:63425 2023-02-11 04:00:37,814 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:00:37,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:37,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:38,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:39,122 INFO [Listener at localhost/33753] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676088016466 2023-02-11 04:00:39,122 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676088016466 2023-02-11 04:00:39,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] procedure2.ProcedureExecutor(1029): Stored pid=188, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:39,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-02-11 04:00:39,126 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088039126"}]},"ts":"1676088039126"} 2023-02-11 04:00:39,127 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676088016466, state=DISABLING in hbase:meta 2023-02-11 04:00:39,130 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676088016466 to state=DISABLING 2023-02-11 04:00:39,130 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=189, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, UNASSIGN}, {pid=190, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, UNASSIGN}, {pid=191, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, UNASSIGN}, {pid=192, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, UNASSIGN}, {pid=193, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, UNASSIGN}, {pid=194, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, UNASSIGN}] 2023-02-11 04:00:39,132 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=190, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, UNASSIGN 2023-02-11 04:00:39,132 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=189, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, UNASSIGN 2023-02-11 04:00:39,132 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=194, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, UNASSIGN 2023-02-11 04:00:39,132 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=193, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, UNASSIGN 2023-02-11 04:00:39,133 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=192, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, UNASSIGN 2023-02-11 04:00:39,133 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=190 updating hbase:meta row=3452669615c81575c52ff96406a41e9e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:39,133 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088039133"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088039133"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088039133"}]},"ts":"1676088039133"} 2023-02-11 04:00:39,133 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=189 updating hbase:meta row=747a6fa324aa972cea6bbd307f3b1539, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:39,133 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=193 updating hbase:meta row=aaa51003a74d8671fa0840bbfcb549ac, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:39,133 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088039133"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088039133"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088039133"}]},"ts":"1676088039133"} 2023-02-11 04:00:39,133 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=192 updating hbase:meta row=e1d0ef015d92b032f67c5052a11865c4, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:39,133 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=194 updating hbase:meta row=d586dffdbc612899aa5c72d99bdc6519, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:39,134 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088039133"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088039133"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088039133"}]},"ts":"1676088039133"} 2023-02-11 04:00:39,134 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088039133"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088039133"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088039133"}]},"ts":"1676088039133"} 2023-02-11 04:00:39,133 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088039133"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088039133"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088039133"}]},"ts":"1676088039133"} 2023-02-11 04:00:39,134 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=195, ppid=190, state=RUNNABLE; CloseRegionProcedure 3452669615c81575c52ff96406a41e9e, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:39,135 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=196, ppid=189, state=RUNNABLE; CloseRegionProcedure 747a6fa324aa972cea6bbd307f3b1539, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:39,136 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=197, ppid=192, state=RUNNABLE; CloseRegionProcedure e1d0ef015d92b032f67c5052a11865c4, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:39,138 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=198, ppid=194, state=RUNNABLE; CloseRegionProcedure d586dffdbc612899aa5c72d99bdc6519, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:39,139 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=199, ppid=193, state=RUNNABLE; CloseRegionProcedure aaa51003a74d8671fa0840bbfcb549ac, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:39,141 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=191, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, UNASSIGN 2023-02-11 04:00:39,143 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=191 updating hbase:meta row=51e4d6a6293430d21cdf75126c54f808, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:39,143 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088039143"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088039143"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088039143"}]},"ts":"1676088039143"} 2023-02-11 04:00:39,144 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=200, ppid=191, state=RUNNABLE; CloseRegionProcedure 51e4d6a6293430d21cdf75126c54f808, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:39,290 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:39,290 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:39,291 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing d586dffdbc612899aa5c72d99bdc6519, disabling compactions & flushes 2023-02-11 04:00:39,292 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e1d0ef015d92b032f67c5052a11865c4, disabling compactions & flushes 2023-02-11 04:00:39,290 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:39,292 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:39,292 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:39,293 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 3452669615c81575c52ff96406a41e9e, disabling compactions & flushes 2023-02-11 04:00:39,292 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:39,293 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:39,293 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:39,293 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:39,293 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. after waiting 0 ms 2023-02-11 04:00:39,293 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. after waiting 0 ms 2023-02-11 04:00:39,293 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. after waiting 0 ms 2023-02-11 04:00:39,293 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:39,293 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:39,293 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:39,300 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-11 04:00:39,301 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-11 04:00:39,301 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-11 04:00:39,301 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4. 2023-02-11 04:00:39,301 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e1d0ef015d92b032f67c5052a11865c4: Waiting for close lock at 1676088039292Running coprocessor pre-close hooks at 1676088039292Disabling compacts and flushes for region at 1676088039292Disabling writes for close at 1676088039293 (+1 ms)Writing region close event to WAL at 1676088039298 (+5 ms)Running coprocessor post-close hooks at 1676088039301 (+3 ms)Closed at 1676088039301 2023-02-11 04:00:39,301 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519. 2023-02-11 04:00:39,301 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for d586dffdbc612899aa5c72d99bdc6519: Waiting for close lock at 1676088039290Running coprocessor pre-close hooks at 1676088039290Disabling compacts and flushes for region at 1676088039290Disabling writes for close at 1676088039293 (+3 ms)Writing region close event to WAL at 1676088039298 (+5 ms)Running coprocessor post-close hooks at 1676088039301 (+3 ms)Closed at 1676088039301 2023-02-11 04:00:39,302 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e. 2023-02-11 04:00:39,302 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 3452669615c81575c52ff96406a41e9e: Waiting for close lock at 1676088039292Running coprocessor pre-close hooks at 1676088039292Disabling compacts and flushes for region at 1676088039292Disabling writes for close at 1676088039293 (+1 ms)Writing region close event to WAL at 1676088039298 (+5 ms)Running coprocessor post-close hooks at 1676088039302 (+4 ms)Closed at 1676088039302 2023-02-11 04:00:39,303 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:39,303 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:39,304 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 747a6fa324aa972cea6bbd307f3b1539, disabling compactions & flushes 2023-02-11 04:00:39,304 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:39,304 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:39,304 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. after waiting 0 ms 2023-02-11 04:00:39,304 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:39,304 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=194 updating hbase:meta row=d586dffdbc612899aa5c72d99bdc6519, regionState=CLOSED 2023-02-11 04:00:39,311 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088039304"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088039304"}]},"ts":"1676088039304"} 2023-02-11 04:00:39,311 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:39,311 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:39,312 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing aaa51003a74d8671fa0840bbfcb549ac, disabling compactions & flushes 2023-02-11 04:00:39,312 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:39,312 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:39,312 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. after waiting 0 ms 2023-02-11 04:00:39,312 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:39,312 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=190 updating hbase:meta row=3452669615c81575c52ff96406a41e9e, regionState=CLOSED 2023-02-11 04:00:39,312 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088039312"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088039312"}]},"ts":"1676088039312"} 2023-02-11 04:00:39,312 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:39,312 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:39,313 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 51e4d6a6293430d21cdf75126c54f808, disabling compactions & flushes 2023-02-11 04:00:39,313 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:39,313 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:39,313 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. after waiting 0 ms 2023-02-11 04:00:39,313 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:39,313 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=192 updating hbase:meta row=e1d0ef015d92b032f67c5052a11865c4, regionState=CLOSED 2023-02-11 04:00:39,313 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088039313"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088039313"}]},"ts":"1676088039313"} 2023-02-11 04:00:39,315 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-11 04:00:39,316 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=198, resume processing ppid=194 2023-02-11 04:00:39,316 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=198, ppid=194, state=SUCCESS; CloseRegionProcedure d586dffdbc612899aa5c72d99bdc6519, server=jenkins-hbase4.apache.org,42385,1676088013771 in 177 msec 2023-02-11 04:00:39,316 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539. 2023-02-11 04:00:39,316 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 747a6fa324aa972cea6bbd307f3b1539: Waiting for close lock at 1676088039303Running coprocessor pre-close hooks at 1676088039303Disabling compacts and flushes for region at 1676088039303Disabling writes for close at 1676088039304 (+1 ms)Writing region close event to WAL at 1676088039311 (+7 ms)Running coprocessor post-close hooks at 1676088039316 (+5 ms)Closed at 1676088039316 2023-02-11 04:00:39,317 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=195, resume processing ppid=190 2023-02-11 04:00:39,317 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=195, ppid=190, state=SUCCESS; CloseRegionProcedure 3452669615c81575c52ff96406a41e9e, server=jenkins-hbase4.apache.org,40829,1676088013739 in 180 msec 2023-02-11 04:00:39,317 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-11 04:00:39,318 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=197, resume processing ppid=192 2023-02-11 04:00:39,318 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-11 04:00:39,318 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=197, ppid=192, state=SUCCESS; CloseRegionProcedure e1d0ef015d92b032f67c5052a11865c4, server=jenkins-hbase4.apache.org,46851,1676088013694 in 179 msec 2023-02-11 04:00:39,318 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:39,318 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808. 2023-02-11 04:00:39,318 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 51e4d6a6293430d21cdf75126c54f808: Waiting for close lock at 1676088039312Running coprocessor pre-close hooks at 1676088039312Disabling compacts and flushes for region at 1676088039312Disabling writes for close at 1676088039313 (+1 ms)Writing region close event to WAL at 1676088039314 (+1 ms)Running coprocessor post-close hooks at 1676088039318 (+4 ms)Closed at 1676088039318 2023-02-11 04:00:39,318 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=194, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=d586dffdbc612899aa5c72d99bdc6519, UNASSIGN in 186 msec 2023-02-11 04:00:39,318 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac. 2023-02-11 04:00:39,318 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=190, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=3452669615c81575c52ff96406a41e9e, UNASSIGN in 187 msec 2023-02-11 04:00:39,318 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for aaa51003a74d8671fa0840bbfcb549ac: Waiting for close lock at 1676088039311Running coprocessor pre-close hooks at 1676088039311Disabling compacts and flushes for region at 1676088039311Disabling writes for close at 1676088039312 (+1 ms)Writing region close event to WAL at 1676088039314 (+2 ms)Running coprocessor post-close hooks at 1676088039318 (+4 ms)Closed at 1676088039318 2023-02-11 04:00:39,319 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=189 updating hbase:meta row=747a6fa324aa972cea6bbd307f3b1539, regionState=CLOSED 2023-02-11 04:00:39,319 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676088039319"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088039319"}]},"ts":"1676088039319"} 2023-02-11 04:00:39,319 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=192, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=e1d0ef015d92b032f67c5052a11865c4, UNASSIGN in 188 msec 2023-02-11 04:00:39,319 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:39,319 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=191 updating hbase:meta row=51e4d6a6293430d21cdf75126c54f808, regionState=CLOSED 2023-02-11 04:00:39,320 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088039319"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088039319"}]},"ts":"1676088039319"} 2023-02-11 04:00:39,320 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:39,320 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=193 updating hbase:meta row=aaa51003a74d8671fa0840bbfcb549ac, regionState=CLOSED 2023-02-11 04:00:39,321 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676088039320"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088039320"}]},"ts":"1676088039320"} 2023-02-11 04:00:39,322 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=196, resume processing ppid=189 2023-02-11 04:00:39,322 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=196, ppid=189, state=SUCCESS; CloseRegionProcedure 747a6fa324aa972cea6bbd307f3b1539, server=jenkins-hbase4.apache.org,42385,1676088013771 in 185 msec 2023-02-11 04:00:39,323 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=200, resume processing ppid=191 2023-02-11 04:00:39,323 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=200, ppid=191, state=SUCCESS; CloseRegionProcedure 51e4d6a6293430d21cdf75126c54f808, server=jenkins-hbase4.apache.org,46851,1676088013694 in 177 msec 2023-02-11 04:00:39,323 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=189, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=747a6fa324aa972cea6bbd307f3b1539, UNASSIGN in 192 msec 2023-02-11 04:00:39,324 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=199, resume processing ppid=193 2023-02-11 04:00:39,324 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=191, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=51e4d6a6293430d21cdf75126c54f808, UNASSIGN in 193 msec 2023-02-11 04:00:39,324 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=199, ppid=193, state=SUCCESS; CloseRegionProcedure aaa51003a74d8671fa0840bbfcb549ac, server=jenkins-hbase4.apache.org,40829,1676088013739 in 183 msec 2023-02-11 04:00:39,325 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=193, resume processing ppid=188 2023-02-11 04:00:39,325 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=193, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676088016466, region=aaa51003a74d8671fa0840bbfcb549ac, UNASSIGN in 194 msec 2023-02-11 04:00:39,325 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088039325"}]},"ts":"1676088039325"} 2023-02-11 04:00:39,326 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676088016466, state=DISABLED in hbase:meta 2023-02-11 04:00:39,328 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676088016466 to state=DISABLED 2023-02-11 04:00:39,329 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=188, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676088016466 in 206 msec 2023-02-11 04:00:39,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-02-11 04:00:39,377 INFO [Listener at localhost/33753] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676088016466, procId: 188 completed 2023-02-11 04:00:39,378 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.HMaster$5(2215): Client=jenkins//172.31.14.131 delete testRestoreSnapshot-1676088016466 2023-02-11 04:00:39,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] procedure2.ProcedureExecutor(1029): Stored pid=201, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:39,380 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=201, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:39,380 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=201, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:39,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=201 2023-02-11 04:00:39,387 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:39,387 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:39,387 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:39,387 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:39,387 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:39,387 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:39,389 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf, FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/recovered.edits] 2023-02-11 04:00:39,389 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf, FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/recovered.edits] 2023-02-11 04:00:39,389 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf, FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/recovered.edits] 2023-02-11 04:00:39,389 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf, FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/recovered.edits] 2023-02-11 04:00:39,389 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf, FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/recovered.edits] 2023-02-11 04:00:39,389 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf, FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/recovered.edits] 2023-02-11 04:00:39,393 ERROR [HFileArchiver-5] backup.HFileArchiver(523): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/testRestoreSnapshot-1676088016466=747a6fa324aa972cea6bbd307f3b1539-0a646c34d29c4963afc688a0da1fd6d4 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/testRestoreSnapshot-1676088016466=747a6fa324aa972cea6bbd307f3b1539-0a646c34d29c4963afc688a0da1fd6d4. archiveLen: 0 currentLen: 0 archiveMtime: 1676088035379 currentMtime: 1676088035749 2023-02-11 04:00:39,393 ERROR [HFileArchiver-6] backup.HFileArchiver(523): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/testRestoreSnapshot-1676088016466=3452669615c81575c52ff96406a41e9e-a11c828e49bd4e7a9cf05be9ff07ad7f already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/testRestoreSnapshot-1676088016466=3452669615c81575c52ff96406a41e9e-a11c828e49bd4e7a9cf05be9ff07ad7f. archiveLen: 0 currentLen: 0 archiveMtime: 1676088035379 currentMtime: 1676088035742 2023-02-11 04:00:39,393 ERROR [HFileArchiver-4] backup.HFileArchiver(523): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/testRestoreSnapshot-1676088016466=51e4d6a6293430d21cdf75126c54f808-1b38064ae3304eca9c176d5669359823 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/testRestoreSnapshot-1676088016466=51e4d6a6293430d21cdf75126c54f808-1b38064ae3304eca9c176d5669359823. archiveLen: 0 currentLen: 0 archiveMtime: 1676088035379 currentMtime: 1676088035738 2023-02-11 04:00:39,393 ERROR [HFileArchiver-1] backup.HFileArchiver(523): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/testRestoreSnapshot-1676088016466=d586dffdbc612899aa5c72d99bdc6519-d1e1bf338c724083a10375a4bd78b7d9 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/testRestoreSnapshot-1676088016466=d586dffdbc612899aa5c72d99bdc6519-d1e1bf338c724083a10375a4bd78b7d9. archiveLen: 0 currentLen: 0 archiveMtime: 1676088035379 currentMtime: 1676088035738 2023-02-11 04:00:39,394 INFO [HFileArchiver-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/testRestoreSnapshot-1676088016466=747a6fa324aa972cea6bbd307f3b1539-0a646c34d29c4963afc688a0da1fd6d4 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/testRestoreSnapshot-1676088016466=747a6fa324aa972cea6bbd307f3b1539-0a646c34d29c4963afc688a0da1fd6d4.1676088039389. 2023-02-11 04:00:39,394 INFO [HFileArchiver-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/testRestoreSnapshot-1676088016466=51e4d6a6293430d21cdf75126c54f808-1b38064ae3304eca9c176d5669359823 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/testRestoreSnapshot-1676088016466=51e4d6a6293430d21cdf75126c54f808-1b38064ae3304eca9c176d5669359823.1676088039389. 2023-02-11 04:00:39,394 ERROR [HFileArchiver-2] backup.HFileArchiver(523): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/testRestoreSnapshot-1676088016466=e1d0ef015d92b032f67c5052a11865c4-93d3b4e383ae4310a66075c83ee5c664 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/testRestoreSnapshot-1676088016466=e1d0ef015d92b032f67c5052a11865c4-93d3b4e383ae4310a66075c83ee5c664. archiveLen: 0 currentLen: 0 archiveMtime: 1676088035380 currentMtime: 1676088035741 2023-02-11 04:00:39,394 ERROR [HFileArchiver-7] backup.HFileArchiver(523): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/testRestoreSnapshot-1676088016466=aaa51003a74d8671fa0840bbfcb549ac-3f27b1f3105b45bf81c17ba9c704bf44 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/testRestoreSnapshot-1676088016466=aaa51003a74d8671fa0840bbfcb549ac-3f27b1f3105b45bf81c17ba9c704bf44. archiveLen: 0 currentLen: 0 archiveMtime: 1676088035379 currentMtime: 1676088035749 2023-02-11 04:00:39,394 INFO [HFileArchiver-6] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/testRestoreSnapshot-1676088016466=3452669615c81575c52ff96406a41e9e-a11c828e49bd4e7a9cf05be9ff07ad7f to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/testRestoreSnapshot-1676088016466=3452669615c81575c52ff96406a41e9e-a11c828e49bd4e7a9cf05be9ff07ad7f.1676088039390. 2023-02-11 04:00:39,394 INFO [HFileArchiver-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/testRestoreSnapshot-1676088016466=d586dffdbc612899aa5c72d99bdc6519-d1e1bf338c724083a10375a4bd78b7d9 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/testRestoreSnapshot-1676088016466=d586dffdbc612899aa5c72d99bdc6519-d1e1bf338c724083a10375a4bd78b7d9.1676088039389. 2023-02-11 04:00:39,394 INFO [HFileArchiver-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/testRestoreSnapshot-1676088016466=e1d0ef015d92b032f67c5052a11865c4-93d3b4e383ae4310a66075c83ee5c664 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/testRestoreSnapshot-1676088016466=e1d0ef015d92b032f67c5052a11865c4-93d3b4e383ae4310a66075c83ee5c664.1676088039389. 2023-02-11 04:00:39,394 INFO [HFileArchiver-7] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/testRestoreSnapshot-1676088016466=aaa51003a74d8671fa0840bbfcb549ac-3f27b1f3105b45bf81c17ba9c704bf44 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/testRestoreSnapshot-1676088016466=aaa51003a74d8671fa0840bbfcb549ac-3f27b1f3105b45bf81c17ba9c704bf44.1676088039389. 2023-02-11 04:00:39,395 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/testRestoreSnapshot-1676088016466=747a6fa324aa972cea6bbd307f3b1539-0a646c34d29c4963afc688a0da1fd6d4 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/testRestoreSnapshot-1676088016466=747a6fa324aa972cea6bbd307f3b1539-0a646c34d29c4963afc688a0da1fd6d4 2023-02-11 04:00:39,395 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/testRestoreSnapshot-1676088016466=3452669615c81575c52ff96406a41e9e-a11c828e49bd4e7a9cf05be9ff07ad7f to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/testRestoreSnapshot-1676088016466=3452669615c81575c52ff96406a41e9e-a11c828e49bd4e7a9cf05be9ff07ad7f 2023-02-11 04:00:39,395 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/testRestoreSnapshot-1676088016466=51e4d6a6293430d21cdf75126c54f808-1b38064ae3304eca9c176d5669359823 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/testRestoreSnapshot-1676088016466=51e4d6a6293430d21cdf75126c54f808-1b38064ae3304eca9c176d5669359823 2023-02-11 04:00:39,395 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/testRestoreSnapshot-1676088016466=d586dffdbc612899aa5c72d99bdc6519-d1e1bf338c724083a10375a4bd78b7d9 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/testRestoreSnapshot-1676088016466=d586dffdbc612899aa5c72d99bdc6519-d1e1bf338c724083a10375a4bd78b7d9 2023-02-11 04:00:39,395 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/testRestoreSnapshot-1676088016466=e1d0ef015d92b032f67c5052a11865c4-93d3b4e383ae4310a66075c83ee5c664 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/testRestoreSnapshot-1676088016466=e1d0ef015d92b032f67c5052a11865c4-93d3b4e383ae4310a66075c83ee5c664 2023-02-11 04:00:39,396 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/testRestoreSnapshot-1676088016466=aaa51003a74d8671fa0840bbfcb549ac-3f27b1f3105b45bf81c17ba9c704bf44 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/testRestoreSnapshot-1676088016466=aaa51003a74d8671fa0840bbfcb549ac-3f27b1f3105b45bf81c17ba9c704bf44 2023-02-11 04:00:39,397 ERROR [HFileArchiver-1] backup.HFileArchiver(523): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/testRestoreSnapshot-1676088016466=d586dffdbc612899aa5c72d99bdc6519-e928d1188fee412e9c504ba31bbb3ef6 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/testRestoreSnapshot-1676088016466=d586dffdbc612899aa5c72d99bdc6519-e928d1188fee412e9c504ba31bbb3ef6. archiveLen: 0 currentLen: 0 archiveMtime: 1676088035381 currentMtime: 1676088035749 2023-02-11 04:00:39,397 ERROR [HFileArchiver-6] backup.HFileArchiver(523): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/testRestoreSnapshot-1676088016466=3452669615c81575c52ff96406a41e9e-f60cf166ab3e4854a8bb6a91f3df546f already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/testRestoreSnapshot-1676088016466=3452669615c81575c52ff96406a41e9e-f60cf166ab3e4854a8bb6a91f3df546f. archiveLen: 0 currentLen: 0 archiveMtime: 1676088035381 currentMtime: 1676088035749 2023-02-11 04:00:39,397 ERROR [HFileArchiver-5] backup.HFileArchiver(523): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/testRestoreSnapshot-1676088016466=747a6fa324aa972cea6bbd307f3b1539-7dc891a823984ca99b9b428443d59643 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/testRestoreSnapshot-1676088016466=747a6fa324aa972cea6bbd307f3b1539-7dc891a823984ca99b9b428443d59643. archiveLen: 0 currentLen: 0 archiveMtime: 1676088035382 currentMtime: 1676088035751 2023-02-11 04:00:39,397 ERROR [HFileArchiver-4] backup.HFileArchiver(523): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/testRestoreSnapshot-1676088016466=51e4d6a6293430d21cdf75126c54f808-c1e521afb6ba4af19d964c416aa5a0de already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/testRestoreSnapshot-1676088016466=51e4d6a6293430d21cdf75126c54f808-c1e521afb6ba4af19d964c416aa5a0de. archiveLen: 0 currentLen: 0 archiveMtime: 1676088035381 currentMtime: 1676088035742 2023-02-11 04:00:39,397 INFO [HFileArchiver-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/testRestoreSnapshot-1676088016466=d586dffdbc612899aa5c72d99bdc6519-e928d1188fee412e9c504ba31bbb3ef6 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/testRestoreSnapshot-1676088016466=d586dffdbc612899aa5c72d99bdc6519-e928d1188fee412e9c504ba31bbb3ef6.1676088039389. 2023-02-11 04:00:39,397 ERROR [HFileArchiver-7] backup.HFileArchiver(523): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/testRestoreSnapshot-1676088016466=aaa51003a74d8671fa0840bbfcb549ac-e6cd8a1c78d94d0c9da6aff3413bb91a already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/testRestoreSnapshot-1676088016466=aaa51003a74d8671fa0840bbfcb549ac-e6cd8a1c78d94d0c9da6aff3413bb91a. archiveLen: 0 currentLen: 0 archiveMtime: 1676088035381 currentMtime: 1676088035751 2023-02-11 04:00:39,398 INFO [HFileArchiver-6] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/testRestoreSnapshot-1676088016466=3452669615c81575c52ff96406a41e9e-f60cf166ab3e4854a8bb6a91f3df546f to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/testRestoreSnapshot-1676088016466=3452669615c81575c52ff96406a41e9e-f60cf166ab3e4854a8bb6a91f3df546f.1676088039390. 2023-02-11 04:00:39,398 INFO [HFileArchiver-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/testRestoreSnapshot-1676088016466=51e4d6a6293430d21cdf75126c54f808-c1e521afb6ba4af19d964c416aa5a0de to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/testRestoreSnapshot-1676088016466=51e4d6a6293430d21cdf75126c54f808-c1e521afb6ba4af19d964c416aa5a0de.1676088039389. 2023-02-11 04:00:39,398 ERROR [HFileArchiver-2] backup.HFileArchiver(523): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/testRestoreSnapshot-1676088016466=e1d0ef015d92b032f67c5052a11865c4-e1f68e07f6354647bc060c949ab8bd60 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/testRestoreSnapshot-1676088016466=e1d0ef015d92b032f67c5052a11865c4-e1f68e07f6354647bc060c949ab8bd60. archiveLen: 0 currentLen: 0 archiveMtime: 1676088035381 currentMtime: 1676088035749 2023-02-11 04:00:39,398 INFO [HFileArchiver-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/testRestoreSnapshot-1676088016466=747a6fa324aa972cea6bbd307f3b1539-7dc891a823984ca99b9b428443d59643 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/testRestoreSnapshot-1676088016466=747a6fa324aa972cea6bbd307f3b1539-7dc891a823984ca99b9b428443d59643.1676088039389. 2023-02-11 04:00:39,398 INFO [HFileArchiver-7] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/testRestoreSnapshot-1676088016466=aaa51003a74d8671fa0840bbfcb549ac-e6cd8a1c78d94d0c9da6aff3413bb91a to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/testRestoreSnapshot-1676088016466=aaa51003a74d8671fa0840bbfcb549ac-e6cd8a1c78d94d0c9da6aff3413bb91a.1676088039389. 2023-02-11 04:00:39,398 INFO [HFileArchiver-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/testRestoreSnapshot-1676088016466=e1d0ef015d92b032f67c5052a11865c4-e1f68e07f6354647bc060c949ab8bd60 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/testRestoreSnapshot-1676088016466=e1d0ef015d92b032f67c5052a11865c4-e1f68e07f6354647bc060c949ab8bd60.1676088039389. 2023-02-11 04:00:39,399 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/testRestoreSnapshot-1676088016466=d586dffdbc612899aa5c72d99bdc6519-e928d1188fee412e9c504ba31bbb3ef6 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/cf/testRestoreSnapshot-1676088016466=d586dffdbc612899aa5c72d99bdc6519-e928d1188fee412e9c504ba31bbb3ef6 2023-02-11 04:00:39,399 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/testRestoreSnapshot-1676088016466=3452669615c81575c52ff96406a41e9e-f60cf166ab3e4854a8bb6a91f3df546f to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/cf/testRestoreSnapshot-1676088016466=3452669615c81575c52ff96406a41e9e-f60cf166ab3e4854a8bb6a91f3df546f 2023-02-11 04:00:39,399 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/testRestoreSnapshot-1676088016466=aaa51003a74d8671fa0840bbfcb549ac-e6cd8a1c78d94d0c9da6aff3413bb91a to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/cf/testRestoreSnapshot-1676088016466=aaa51003a74d8671fa0840bbfcb549ac-e6cd8a1c78d94d0c9da6aff3413bb91a 2023-02-11 04:00:39,399 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/testRestoreSnapshot-1676088016466=51e4d6a6293430d21cdf75126c54f808-c1e521afb6ba4af19d964c416aa5a0de to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/cf/testRestoreSnapshot-1676088016466=51e4d6a6293430d21cdf75126c54f808-c1e521afb6ba4af19d964c416aa5a0de 2023-02-11 04:00:39,399 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/testRestoreSnapshot-1676088016466=e1d0ef015d92b032f67c5052a11865c4-e1f68e07f6354647bc060c949ab8bd60 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/cf/testRestoreSnapshot-1676088016466=e1d0ef015d92b032f67c5052a11865c4-e1f68e07f6354647bc060c949ab8bd60 2023-02-11 04:00:39,400 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/testRestoreSnapshot-1676088016466=747a6fa324aa972cea6bbd307f3b1539-7dc891a823984ca99b9b428443d59643 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/cf/testRestoreSnapshot-1676088016466=747a6fa324aa972cea6bbd307f3b1539-7dc891a823984ca99b9b428443d59643 2023-02-11 04:00:39,402 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/recovered.edits/18.seqid to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac/recovered.edits/18.seqid 2023-02-11 04:00:39,403 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/recovered.edits/18.seqid to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519/recovered.edits/18.seqid 2023-02-11 04:00:39,403 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/recovered.edits/18.seqid to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4/recovered.edits/18.seqid 2023-02-11 04:00:39,403 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/recovered.edits/18.seqid to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808/recovered.edits/18.seqid 2023-02-11 04:00:39,403 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/recovered.edits/18.seqid to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e/recovered.edits/18.seqid 2023-02-11 04:00:39,403 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/aaa51003a74d8671fa0840bbfcb549ac 2023-02-11 04:00:39,403 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/d586dffdbc612899aa5c72d99bdc6519 2023-02-11 04:00:39,404 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/e1d0ef015d92b032f67c5052a11865c4 2023-02-11 04:00:39,404 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/51e4d6a6293430d21cdf75126c54f808 2023-02-11 04:00:39,404 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/3452669615c81575c52ff96406a41e9e 2023-02-11 04:00:39,404 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/recovered.edits/18.seqid to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539/recovered.edits/18.seqid 2023-02-11 04:00:39,405 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testRestoreSnapshot-1676088016466/747a6fa324aa972cea6bbd307f3b1539 2023-02-11 04:00:39,405 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676088016466 regions 2023-02-11 04:00:39,405 DEBUG [PEWorker-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f 2023-02-11 04:00:39,405 DEBUG [PEWorker-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf] 2023-02-11 04:00:39,407 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f. archiveLen: 0 currentLen: 0 archiveMtime: 1676088035391 currentMtime: 1676088035664 2023-02-11 04:00:39,408 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f.1676088039405. 2023-02-11 04:00:39,408 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-a87ff679a2f3e71d9181a67b7542122c2023021121a7b039d1204dde9586e0349b483a2f 2023-02-11 04:00:39,409 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2. archiveLen: 0 currentLen: 0 archiveMtime: 1676088035392 currentMtime: 1676088035666 2023-02-11 04:00:39,410 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2.1676088039405. 2023-02-11 04:00:39,410 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-a87ff679a2f3e71d9181a67b7542122c20230211cb4c1343d6554782a4d07ff20e0869e2 2023-02-11 04:00:39,411 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0. archiveLen: 0 currentLen: 0 archiveMtime: 1676088035393 currentMtime: 1676088035668 2023-02-11 04:00:39,412 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0.1676088039405. 2023-02-11 04:00:39,412 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-c4ca4238a0b923820dcc509a6f75849b2023021145afa94e6db94cbf8c2b4a5626a8b2e0 2023-02-11 04:00:39,413 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7. archiveLen: 0 currentLen: 0 archiveMtime: 1676088035394 currentMtime: 1676088035670 2023-02-11 04:00:39,414 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7.1676088039405. 2023-02-11 04:00:39,414 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-c4ca4238a0b923820dcc509a6f75849b2023021175de917790b84b0a8df11ec8437717d7 2023-02-11 04:00:39,415 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15. archiveLen: 0 currentLen: 0 archiveMtime: 1676088035395 currentMtime: 1676088035672 2023-02-11 04:00:39,416 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15.1676088039405. 2023-02-11 04:00:39,416 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-c81e728d9d4c2f636f067f89cc14862c202302116ccfc1d24e8243188ef42a2c5abbdc15 2023-02-11 04:00:39,417 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806. archiveLen: 0 currentLen: 0 archiveMtime: 1676088035397 currentMtime: 1676088035674 2023-02-11 04:00:39,417 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806.1676088039405. 2023-02-11 04:00:39,418 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-c81e728d9d4c2f636f067f89cc14862c20230211ac0f72aacddf4ad989a1e78edf656806 2023-02-11 04:00:39,419 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78. archiveLen: 0 currentLen: 0 archiveMtime: 1676088035398 currentMtime: 1676088035676 2023-02-11 04:00:39,419 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78.1676088039405. 2023-02-11 04:00:39,420 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-d41d8cd98f00b204e9800998ecf8427e2023021131ace875210d45d9af554bf652803f78 2023-02-11 04:00:39,421 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b. archiveLen: 0 currentLen: 0 archiveMtime: 1676088035399 currentMtime: 1676088035678 2023-02-11 04:00:39,421 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b.1676088039405. 2023-02-11 04:00:39,422 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-d41d8cd98f00b204e9800998ecf8427e202302115f186de0aec442609bb60523bb47106b 2023-02-11 04:00:39,423 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3. archiveLen: 0 currentLen: 0 archiveMtime: 1676088035400 currentMtime: 1676088035681 2023-02-11 04:00:39,423 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3.1676088039405. 2023-02-11 04:00:39,424 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-e4da3b7fbbce2345d7772b0674a318d520230211257331ff330c4e6e9e228c99446d85d3 2023-02-11 04:00:39,424 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e. archiveLen: 0 currentLen: 0 archiveMtime: 1676088035401 currentMtime: 1676088035684 2023-02-11 04:00:39,425 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e.1676088039405. 2023-02-11 04:00:39,425 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-e4da3b7fbbce2345d7772b0674a318d52023021132aa4d15f9c14beb84c0c30160f1276e 2023-02-11 04:00:39,426 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8. archiveLen: 0 currentLen: 0 archiveMtime: 1676088035402 currentMtime: 1676088035687 2023-02-11 04:00:39,427 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8.1676088039405. 2023-02-11 04:00:39,427 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-eccbc87e4b5ce2fe28308fd9f2a7baf32023021142ff720a170d421587be1646ffb324a8 2023-02-11 04:00:39,428 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45. archiveLen: 0 currentLen: 0 archiveMtime: 1676088035403 currentMtime: 1676088035689 2023-02-11 04:00:39,429 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45.1676088039405. 2023-02-11 04:00:39,429 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f/cf/testRestoreSnapshot-1676088016466=089c12f910ab0d90d02340003e54031f-eccbc87e4b5ce2fe28308fd9f2a7baf320230211867f264e3050468cbb8ab44838417b45 2023-02-11 04:00:39,430 DEBUG [PEWorker-3] backup.HFileArchiver(596): Deleted hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testRestoreSnapshot-1676088016466/089c12f910ab0d90d02340003e54031f 2023-02-11 04:00:39,431 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=201, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:39,434 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testRestoreSnapshot-1676088016466 from hbase:meta 2023-02-11 04:00:39,436 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'testRestoreSnapshot-1676088016466' descriptor. 2023-02-11 04:00:39,436 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=201, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:39,436 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'testRestoreSnapshot-1676088016466' from region states. 2023-02-11 04:00:39,437 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088039437"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:39,437 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088039437"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:39,437 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088039437"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:39,437 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088039437"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:39,437 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088039437"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:39,437 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088039437"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:39,438 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-11 04:00:39,438 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 747a6fa324aa972cea6bbd307f3b1539, NAME => 'testRestoreSnapshot-1676088016466,,1676088016488.747a6fa324aa972cea6bbd307f3b1539.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 3452669615c81575c52ff96406a41e9e, NAME => 'testRestoreSnapshot-1676088016466,1,1676088016488.3452669615c81575c52ff96406a41e9e.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 51e4d6a6293430d21cdf75126c54f808, NAME => 'testRestoreSnapshot-1676088016466,2,1676088016488.51e4d6a6293430d21cdf75126c54f808.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => e1d0ef015d92b032f67c5052a11865c4, NAME => 'testRestoreSnapshot-1676088016466,3,1676088016488.e1d0ef015d92b032f67c5052a11865c4.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => aaa51003a74d8671fa0840bbfcb549ac, NAME => 'testRestoreSnapshot-1676088016466,4,1676088016488.aaa51003a74d8671fa0840bbfcb549ac.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => d586dffdbc612899aa5c72d99bdc6519, NAME => 'testRestoreSnapshot-1676088016466,5,1676088016488.d586dffdbc612899aa5c72d99bdc6519.', STARTKEY => '5', ENDKEY => ''}] 2023-02-11 04:00:39,438 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'testRestoreSnapshot-1676088016466' as deleted. 2023-02-11 04:00:39,439 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676088016466","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676088039438"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:39,440 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table testRestoreSnapshot-1676088016466 state from META 2023-02-11 04:00:39,441 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=201, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676088016466 2023-02-11 04:00:39,442 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=201, state=SUCCESS; DeleteTableProcedure table=testRestoreSnapshot-1676088016466 in 63 msec 2023-02-11 04:00:39,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=201 2023-02-11 04:00:39,631 INFO [Listener at localhost/33753] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRestoreSnapshot-1676088016466, procId: 201 completed 2023-02-11 04:00:39,637 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "emptySnaptb-1676088016466" 2023-02-11 04:00:39,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(333): Deleting snapshot: emptySnaptb-1676088016466 2023-02-11 04:00:39,639 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "snaptb0-1676088016466" 2023-02-11 04:00:39,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(333): Deleting snapshot: snaptb0-1676088016466 2023-02-11 04:00:39,642 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "snaptb1-1676088016466" 2023-02-11 04:00:39,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(333): Deleting snapshot: snaptb1-1676088016466 2023-02-11 04:00:39,669 INFO [Listener at localhost/33753] hbase.ResourceChecker(175): after: client.TestMobRestoreSnapshotFromClientSimple#testRestoreSnapshot Thread=424 (was 358) Potentially hanging thread: hconnection-0x48801dbb-shared-pool-26 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1342560069_15 at /127.0.0.1:52834 [Waiting for operation #6] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/cluster_b673684c-6f28-f6ab-2a99-eadf0e1a7de2/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1459830812_15 at /127.0.0.1:49814 [Waiting for operation #250] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_940823654_15 at /127.0.0.1:50294 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x594888d9-metaLookup-shared--pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x594888d9-metaLookup-shared--pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1459830812_15 at /127.0.0.1:57640 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-14 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: HFileArchiver-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-112778749_15 at /127.0.0.1:45214 [Waiting for operation #605] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_940823654_15 at /127.0.0.1:53388 [Waiting for operation #743] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1342560069_15 at /127.0.0.1:50308 [Waiting for operation #496] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x56d8d559-shared-pool-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-112778749_15 at /127.0.0.1:57672 [Waiting for operation #515] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x594888d9-shared-pool-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x594888d9-shared-pool-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x48801dbb-shared-pool-27 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:63425@0x69705b98-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/cluster_b673684c-6f28-f6ab-2a99-eadf0e1a7de2/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:63425@0x69705b98 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$45/1084683295.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_940823654_15 at /127.0.0.1:45210 [Waiting for operation #603] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-112778749_15 at /127.0.0.1:50320 [Waiting for operation #525] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1459830812_15 at /127.0.0.1:50290 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-16 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/cluster_b673684c-6f28-f6ab-2a99-eadf0e1a7de2/dfs/data/data4/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/cluster_b673684c-6f28-f6ab-2a99-eadf0e1a7de2/dfs/data/data5/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/cluster_b673684c-6f28-f6ab-2a99-eadf0e1a7de2/dfs/data/data6/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-15 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x594888d9-shared-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x594888d9-metaLookup-shared--pool-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x594888d9-shared-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x594888d9-metaLookup-shared--pool-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x48801dbb-shared-pool-25 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-112778749_15 at /127.0.0.1:44966 [Waiting for operation #12] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_940823654_15 at /127.0.0.1:57628 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1342560069_15 at /127.0.0.1:57656 [Waiting for operation #537] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:63425@0x69705b98-SendThread(127.0.0.1:63425) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: hconnection-0x594888d9-shared-pool-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-112778749_15 at /127.0.0.1:57622 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/cluster_b673684c-6f28-f6ab-2a99-eadf0e1a7de2/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1459830812_15 at /127.0.0.1:50302 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x48801dbb-shared-pool-28 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=827 (was 691) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=235 (was 223) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 170), AvailableMemoryMB=2748 (was 3209) 2023-02-11 04:00:39,685 INFO [Listener at localhost/33753] hbase.ResourceChecker(147): before: client.TestMobRestoreSnapshotFromClientSimple#testCorruptedSnapshot Thread=424, OpenFileDescriptor=827, MaxFileDescriptor=60000, SystemLoadAverage=235, ProcessCount=169, AvailableMemoryMB=2748 2023-02-11 04:00:39,687 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'testCorruptedSnapshot-1676088039685', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2023-02-11 04:00:39,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] procedure2.ProcedureExecutor(1029): Stored pid=202, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testCorruptedSnapshot-1676088039685 2023-02-11 04:00:39,689 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676088039685 execute state=CREATE_TABLE_PRE_OPERATION 2023-02-11 04:00:39,690 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "testCorruptedSnapshot-1676088039685" procId is: 202 2023-02-11 04:00:39,690 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676088039685 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-11 04:00:39,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-11 04:00:39,695 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56 2023-02-11 04:00:39,695 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413 2023-02-11 04:00:39,695 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca 2023-02-11 04:00:39,695 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8 2023-02-11 04:00:39,695 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f 2023-02-11 04:00:39,695 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351 2023-02-11 04:00:39,696 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56 empty. 2023-02-11 04:00:39,696 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413 empty. 2023-02-11 04:00:39,696 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca empty. 2023-02-11 04:00:39,696 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8 empty. 2023-02-11 04:00:39,696 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56 2023-02-11 04:00:39,696 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351 empty. 2023-02-11 04:00:39,696 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca 2023-02-11 04:00:39,696 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f empty. 2023-02-11 04:00:39,696 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413 2023-02-11 04:00:39,697 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8 2023-02-11 04:00:39,697 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351 2023-02-11 04:00:39,697 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f 2023-02-11 04:00:39,697 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testCorruptedSnapshot-1676088039685 regions 2023-02-11 04:00:39,714 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/.tabledesc/.tableinfo.0000000001 2023-02-11 04:00:39,716 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-2] regionserver.HRegion(7671): creating {ENCODED => 660b743875c88bf2eaa58aab9da56413, NAME => 'testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testCorruptedSnapshot-1676088039685', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp 2023-02-11 04:00:39,716 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-1] regionserver.HRegion(7671): creating {ENCODED => a388dd856b0a069ab94fe6f6b9fb6351, NAME => 'testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testCorruptedSnapshot-1676088039685', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp 2023-02-11 04:00:39,716 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-0] regionserver.HRegion(7671): creating {ENCODED => 73f6bdfd53354bbf7c9d40fd769c9e56, NAME => 'testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testCorruptedSnapshot-1676088039685', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp 2023-02-11 04:00:39,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-11 04:00:40,141 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-1] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:40,141 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-2] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:40,141 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-1] regionserver.HRegion(1603): Closing a388dd856b0a069ab94fe6f6b9fb6351, disabling compactions & flushes 2023-02-11 04:00:40,141 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-2] regionserver.HRegion(1603): Closing 660b743875c88bf2eaa58aab9da56413, disabling compactions & flushes 2023-02-11 04:00:40,141 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-1] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351. 2023-02-11 04:00:40,141 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-2] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413. 2023-02-11 04:00:40,141 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413. 2023-02-11 04:00:40,141 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351. 2023-02-11 04:00:40,141 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-2] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413. after waiting 0 ms 2023-02-11 04:00:40,141 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-1] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351. after waiting 0 ms 2023-02-11 04:00:40,141 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-1] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351. 2023-02-11 04:00:40,141 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-2] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413. 2023-02-11 04:00:40,141 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-1] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351. 2023-02-11 04:00:40,141 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-2] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413. 2023-02-11 04:00:40,141 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-1] regionserver.HRegion(1557): Region close journal for a388dd856b0a069ab94fe6f6b9fb6351: Waiting for close lock at 1676088040141Disabling compacts and flushes for region at 1676088040141Disabling writes for close at 1676088040141Writing region close event to WAL at 1676088040141Closed at 1676088040141 2023-02-11 04:00:40,141 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-2] regionserver.HRegion(1557): Region close journal for 660b743875c88bf2eaa58aab9da56413: Waiting for close lock at 1676088040141Disabling compacts and flushes for region at 1676088040141Disabling writes for close at 1676088040141Writing region close event to WAL at 1676088040141Closed at 1676088040141 2023-02-11 04:00:40,142 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-1] regionserver.HRegion(7671): creating {ENCODED => ee86f4ea9182f5cda2013084d379724f, NAME => 'testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testCorruptedSnapshot-1676088039685', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp 2023-02-11 04:00:40,142 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-2] regionserver.HRegion(7671): creating {ENCODED => 393f61e585ea6ccae9181b305adcbdf8, NAME => 'testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testCorruptedSnapshot-1676088039685', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp 2023-02-11 04:00:40,142 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:40,142 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-0] regionserver.HRegion(1603): Closing 73f6bdfd53354bbf7c9d40fd769c9e56, disabling compactions & flushes 2023-02-11 04:00:40,142 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. 2023-02-11 04:00:40,143 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. 2023-02-11 04:00:40,143 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. after waiting 0 ms 2023-02-11 04:00:40,143 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. 2023-02-11 04:00:40,143 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. 2023-02-11 04:00:40,143 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-0] regionserver.HRegion(1557): Region close journal for 73f6bdfd53354bbf7c9d40fd769c9e56: Waiting for close lock at 1676088040142Disabling compacts and flushes for region at 1676088040142Disabling writes for close at 1676088040143 (+1 ms)Writing region close event to WAL at 1676088040143Closed at 1676088040143 2023-02-11 04:00:40,143 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-0] regionserver.HRegion(7671): creating {ENCODED => 998536da7be614a24a93f9e7ec31dbca, NAME => 'testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testCorruptedSnapshot-1676088039685', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp 2023-02-11 04:00:40,159 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:40,159 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-0] regionserver.HRegion(1603): Closing 998536da7be614a24a93f9e7ec31dbca, disabling compactions & flushes 2023-02-11 04:00:40,159 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca. 2023-02-11 04:00:40,159 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca. 2023-02-11 04:00:40,159 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca. after waiting 0 ms 2023-02-11 04:00:40,159 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca. 2023-02-11 04:00:40,159 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca. 2023-02-11 04:00:40,159 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-0] regionserver.HRegion(1557): Region close journal for 998536da7be614a24a93f9e7ec31dbca: Waiting for close lock at 1676088040159Disabling compacts and flushes for region at 1676088040159Disabling writes for close at 1676088040159Writing region close event to WAL at 1676088040159Closed at 1676088040159 2023-02-11 04:00:40,159 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-2] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:40,159 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-2] regionserver.HRegion(1603): Closing 393f61e585ea6ccae9181b305adcbdf8, disabling compactions & flushes 2023-02-11 04:00:40,159 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-2] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8. 2023-02-11 04:00:40,159 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8. 2023-02-11 04:00:40,159 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-2] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8. after waiting 0 ms 2023-02-11 04:00:40,159 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-2] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8. 2023-02-11 04:00:40,159 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-2] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8. 2023-02-11 04:00:40,159 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-2] regionserver.HRegion(1557): Region close journal for 393f61e585ea6ccae9181b305adcbdf8: Waiting for close lock at 1676088040159Disabling compacts and flushes for region at 1676088040159Disabling writes for close at 1676088040159Writing region close event to WAL at 1676088040159Closed at 1676088040159 2023-02-11 04:00:40,162 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-1] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:40,162 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-1] regionserver.HRegion(1603): Closing ee86f4ea9182f5cda2013084d379724f, disabling compactions & flushes 2023-02-11 04:00:40,162 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-1] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f. 2023-02-11 04:00:40,162 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f. 2023-02-11 04:00:40,162 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-1] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f. after waiting 0 ms 2023-02-11 04:00:40,162 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-1] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f. 2023-02-11 04:00:40,162 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-1] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f. 2023-02-11 04:00:40,162 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676088039685-pool-1] regionserver.HRegion(1557): Region close journal for ee86f4ea9182f5cda2013084d379724f: Waiting for close lock at 1676088040162Disabling compacts and flushes for region at 1676088040162Disabling writes for close at 1676088040162Writing region close event to WAL at 1676088040162Closed at 1676088040162 2023-02-11 04:00:40,164 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676088039685 execute state=CREATE_TABLE_ADD_TO_META 2023-02-11 04:00:40,165 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088040165"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088040165"}]},"ts":"1676088040165"} 2023-02-11 04:00:40,165 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088040165"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088040165"}]},"ts":"1676088040165"} 2023-02-11 04:00:40,165 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676088040165"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088040165"}]},"ts":"1676088040165"} 2023-02-11 04:00:40,165 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676088040165"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088040165"}]},"ts":"1676088040165"} 2023-02-11 04:00:40,165 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088040165"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088040165"}]},"ts":"1676088040165"} 2023-02-11 04:00:40,165 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088040165"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088040165"}]},"ts":"1676088040165"} 2023-02-11 04:00:40,169 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-11 04:00:40,170 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676088039685 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-11 04:00:40,170 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676088039685","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088040170"}]},"ts":"1676088040170"} 2023-02-11 04:00:40,171 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676088039685, state=ENABLING in hbase:meta 2023-02-11 04:00:40,176 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 04:00:40,177 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 04:00:40,178 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 04:00:40,178 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 04:00:40,178 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 04:00:40,178 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=73f6bdfd53354bbf7c9d40fd769c9e56, ASSIGN}, {pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=a388dd856b0a069ab94fe6f6b9fb6351, ASSIGN}, {pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=660b743875c88bf2eaa58aab9da56413, ASSIGN}, {pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=ee86f4ea9182f5cda2013084d379724f, ASSIGN}, {pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=393f61e585ea6ccae9181b305adcbdf8, ASSIGN}, {pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=998536da7be614a24a93f9e7ec31dbca, ASSIGN}] 2023-02-11 04:00:40,180 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=ee86f4ea9182f5cda2013084d379724f, ASSIGN 2023-02-11 04:00:40,180 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=393f61e585ea6ccae9181b305adcbdf8, ASSIGN 2023-02-11 04:00:40,180 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=660b743875c88bf2eaa58aab9da56413, ASSIGN 2023-02-11 04:00:40,180 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=a388dd856b0a069ab94fe6f6b9fb6351, ASSIGN 2023-02-11 04:00:40,180 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=ee86f4ea9182f5cda2013084d379724f, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40829,1676088013739; forceNewPlan=false, retain=false 2023-02-11 04:00:40,181 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=393f61e585ea6ccae9181b305adcbdf8, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46851,1676088013694; forceNewPlan=false, retain=false 2023-02-11 04:00:40,181 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=660b743875c88bf2eaa58aab9da56413, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46851,1676088013694; forceNewPlan=false, retain=false 2023-02-11 04:00:40,181 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=998536da7be614a24a93f9e7ec31dbca, ASSIGN 2023-02-11 04:00:40,181 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=a388dd856b0a069ab94fe6f6b9fb6351, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42385,1676088013771; forceNewPlan=false, retain=false 2023-02-11 04:00:40,181 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=73f6bdfd53354bbf7c9d40fd769c9e56, ASSIGN 2023-02-11 04:00:40,181 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=998536da7be614a24a93f9e7ec31dbca, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42385,1676088013771; forceNewPlan=false, retain=false 2023-02-11 04:00:40,182 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=73f6bdfd53354bbf7c9d40fd769c9e56, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40829,1676088013739; forceNewPlan=false, retain=false 2023-02-11 04:00:40,330 INFO [jenkins-hbase4:41713] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-11 04:00:40,334 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=205 updating hbase:meta row=660b743875c88bf2eaa58aab9da56413, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:40,334 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=206 updating hbase:meta row=ee86f4ea9182f5cda2013084d379724f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:40,334 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088040334"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088040334"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088040334"}]},"ts":"1676088040334"} 2023-02-11 04:00:40,334 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=208 updating hbase:meta row=998536da7be614a24a93f9e7ec31dbca, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:40,334 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=207 updating hbase:meta row=393f61e585ea6ccae9181b305adcbdf8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:40,335 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676088040334"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088040334"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088040334"}]},"ts":"1676088040334"} 2023-02-11 04:00:40,335 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088040334"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088040334"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088040334"}]},"ts":"1676088040334"} 2023-02-11 04:00:40,334 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088040334"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088040334"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088040334"}]},"ts":"1676088040334"} 2023-02-11 04:00:40,334 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=204 updating hbase:meta row=a388dd856b0a069ab94fe6f6b9fb6351, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:40,335 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088040334"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088040334"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088040334"}]},"ts":"1676088040334"} 2023-02-11 04:00:40,336 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=209, ppid=205, state=RUNNABLE; OpenRegionProcedure 660b743875c88bf2eaa58aab9da56413, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:40,336 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=210, ppid=208, state=RUNNABLE; OpenRegionProcedure 998536da7be614a24a93f9e7ec31dbca, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:40,337 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=211, ppid=207, state=RUNNABLE; OpenRegionProcedure 393f61e585ea6ccae9181b305adcbdf8, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:40,338 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=212, ppid=206, state=RUNNABLE; OpenRegionProcedure ee86f4ea9182f5cda2013084d379724f, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:40,338 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=213, ppid=204, state=RUNNABLE; OpenRegionProcedure a388dd856b0a069ab94fe6f6b9fb6351, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:40,339 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=203 updating hbase:meta row=73f6bdfd53354bbf7c9d40fd769c9e56, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:40,339 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676088040339"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088040339"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088040339"}]},"ts":"1676088040339"} 2023-02-11 04:00:40,340 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=214, ppid=203, state=RUNNABLE; OpenRegionProcedure 73f6bdfd53354bbf7c9d40fd769c9e56, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:40,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-11 04:00:40,490 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8. 2023-02-11 04:00:40,490 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 393f61e585ea6ccae9181b305adcbdf8, NAME => 'testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 04:00:40,490 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676088039685 393f61e585ea6ccae9181b305adcbdf8 2023-02-11 04:00:40,490 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:40,491 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 393f61e585ea6ccae9181b305adcbdf8 2023-02-11 04:00:40,491 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 393f61e585ea6ccae9181b305adcbdf8 2023-02-11 04:00:40,491 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f. 2023-02-11 04:00:40,491 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ee86f4ea9182f5cda2013084d379724f, NAME => 'testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 04:00:40,491 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676088039685 ee86f4ea9182f5cda2013084d379724f 2023-02-11 04:00:40,491 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:40,491 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for ee86f4ea9182f5cda2013084d379724f 2023-02-11 04:00:40,491 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for ee86f4ea9182f5cda2013084d379724f 2023-02-11 04:00:40,491 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351. 2023-02-11 04:00:40,491 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a388dd856b0a069ab94fe6f6b9fb6351, NAME => 'testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 04:00:40,491 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676088039685 a388dd856b0a069ab94fe6f6b9fb6351 2023-02-11 04:00:40,492 INFO [StoreOpener-393f61e585ea6ccae9181b305adcbdf8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 393f61e585ea6ccae9181b305adcbdf8 2023-02-11 04:00:40,492 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:40,492 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for a388dd856b0a069ab94fe6f6b9fb6351 2023-02-11 04:00:40,492 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for a388dd856b0a069ab94fe6f6b9fb6351 2023-02-11 04:00:40,492 INFO [StoreOpener-ee86f4ea9182f5cda2013084d379724f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ee86f4ea9182f5cda2013084d379724f 2023-02-11 04:00:40,493 INFO [StoreOpener-a388dd856b0a069ab94fe6f6b9fb6351-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a388dd856b0a069ab94fe6f6b9fb6351 2023-02-11 04:00:40,493 DEBUG [StoreOpener-393f61e585ea6ccae9181b305adcbdf8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8/cf 2023-02-11 04:00:40,493 DEBUG [StoreOpener-393f61e585ea6ccae9181b305adcbdf8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8/cf 2023-02-11 04:00:40,493 DEBUG [StoreOpener-ee86f4ea9182f5cda2013084d379724f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f/cf 2023-02-11 04:00:40,493 DEBUG [StoreOpener-ee86f4ea9182f5cda2013084d379724f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f/cf 2023-02-11 04:00:40,493 INFO [StoreOpener-393f61e585ea6ccae9181b305adcbdf8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 393f61e585ea6ccae9181b305adcbdf8 columnFamilyName cf 2023-02-11 04:00:40,494 INFO [StoreOpener-ee86f4ea9182f5cda2013084d379724f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ee86f4ea9182f5cda2013084d379724f columnFamilyName cf 2023-02-11 04:00:40,494 DEBUG [StoreOpener-a388dd856b0a069ab94fe6f6b9fb6351-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351/cf 2023-02-11 04:00:40,494 DEBUG [StoreOpener-a388dd856b0a069ab94fe6f6b9fb6351-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351/cf 2023-02-11 04:00:40,494 INFO [StoreOpener-a388dd856b0a069ab94fe6f6b9fb6351-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a388dd856b0a069ab94fe6f6b9fb6351 columnFamilyName cf 2023-02-11 04:00:40,495 INFO [StoreOpener-ee86f4ea9182f5cda2013084d379724f-1] regionserver.HStore(310): Store=ee86f4ea9182f5cda2013084d379724f/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:40,495 INFO [StoreOpener-393f61e585ea6ccae9181b305adcbdf8-1] regionserver.HStore(310): Store=393f61e585ea6ccae9181b305adcbdf8/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:40,495 INFO [StoreOpener-a388dd856b0a069ab94fe6f6b9fb6351-1] regionserver.HStore(310): Store=a388dd856b0a069ab94fe6f6b9fb6351/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:40,496 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f 2023-02-11 04:00:40,496 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8 2023-02-11 04:00:40,496 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351 2023-02-11 04:00:40,496 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f 2023-02-11 04:00:40,496 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8 2023-02-11 04:00:40,496 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351 2023-02-11 04:00:40,498 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 393f61e585ea6ccae9181b305adcbdf8 2023-02-11 04:00:40,499 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for ee86f4ea9182f5cda2013084d379724f 2023-02-11 04:00:40,499 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for a388dd856b0a069ab94fe6f6b9fb6351 2023-02-11 04:00:40,500 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 04:00:40,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 04:00:40,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 04:00:40,501 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened ee86f4ea9182f5cda2013084d379724f; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62434748, jitterRate=-0.06964975595474243}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:40,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for ee86f4ea9182f5cda2013084d379724f: Running coprocessor pre-open hook at 1676088040491Writing region info on filesystem at 1676088040491Initializing all the Stores at 1676088040492 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088040492Cleaning up temporary data from old regions at 1676088040496 (+4 ms)Cleaning up detritus from prior splits at 1676088040497 (+1 ms)Running coprocessor post-open hooks at 1676088040501 (+4 ms)Region opened successfully at 1676088040501 2023-02-11 04:00:40,501 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 393f61e585ea6ccae9181b305adcbdf8; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64915219, jitterRate=-0.03268785774707794}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:40,501 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened a388dd856b0a069ab94fe6f6b9fb6351; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64002836, jitterRate=-0.04628342390060425}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:40,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 393f61e585ea6ccae9181b305adcbdf8: Running coprocessor pre-open hook at 1676088040491Writing region info on filesystem at 1676088040491Initializing all the Stores at 1676088040491Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088040491Cleaning up temporary data from old regions at 1676088040496 (+5 ms)Cleaning up detritus from prior splits at 1676088040497 (+1 ms)Running coprocessor post-open hooks at 1676088040501 (+4 ms)Region opened successfully at 1676088040501 2023-02-11 04:00:40,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for a388dd856b0a069ab94fe6f6b9fb6351: Running coprocessor pre-open hook at 1676088040492Writing region info on filesystem at 1676088040492Initializing all the Stores at 1676088040493 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088040493Cleaning up temporary data from old regions at 1676088040497 (+4 ms)Cleaning up detritus from prior splits at 1676088040497Running coprocessor post-open hooks at 1676088040501 (+4 ms)Region opened successfully at 1676088040501 2023-02-11 04:00:40,502 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f., pid=212, masterSystemTime=1676088040489 2023-02-11 04:00:40,502 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351., pid=213, masterSystemTime=1676088040489 2023-02-11 04:00:40,502 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8., pid=211, masterSystemTime=1676088040487 2023-02-11 04:00:40,503 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351. 2023-02-11 04:00:40,504 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351. 2023-02-11 04:00:40,504 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca. 2023-02-11 04:00:40,504 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 998536da7be614a24a93f9e7ec31dbca, NAME => 'testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 04:00:40,504 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=204 updating hbase:meta row=a388dd856b0a069ab94fe6f6b9fb6351, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:40,504 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676088039685 998536da7be614a24a93f9e7ec31dbca 2023-02-11 04:00:40,504 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088040504"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088040504"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088040504"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088040504"}]},"ts":"1676088040504"} 2023-02-11 04:00:40,504 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f. 2023-02-11 04:00:40,504 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:40,504 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f. 2023-02-11 04:00:40,505 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 998536da7be614a24a93f9e7ec31dbca 2023-02-11 04:00:40,505 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. 2023-02-11 04:00:40,505 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 998536da7be614a24a93f9e7ec31dbca 2023-02-11 04:00:40,505 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 73f6bdfd53354bbf7c9d40fd769c9e56, NAME => 'testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 04:00:40,505 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=206 updating hbase:meta row=ee86f4ea9182f5cda2013084d379724f, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:40,505 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8. 2023-02-11 04:00:40,505 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676088039685 73f6bdfd53354bbf7c9d40fd769c9e56 2023-02-11 04:00:40,505 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:40,505 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088040505"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088040505"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088040505"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088040505"}]},"ts":"1676088040505"} 2023-02-11 04:00:40,505 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8. 2023-02-11 04:00:40,505 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 73f6bdfd53354bbf7c9d40fd769c9e56 2023-02-11 04:00:40,505 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 73f6bdfd53354bbf7c9d40fd769c9e56 2023-02-11 04:00:40,505 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413. 2023-02-11 04:00:40,506 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 660b743875c88bf2eaa58aab9da56413, NAME => 'testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 04:00:40,506 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676088039685 660b743875c88bf2eaa58aab9da56413 2023-02-11 04:00:40,506 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:40,506 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 660b743875c88bf2eaa58aab9da56413 2023-02-11 04:00:40,506 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 660b743875c88bf2eaa58aab9da56413 2023-02-11 04:00:40,506 INFO [StoreOpener-998536da7be614a24a93f9e7ec31dbca-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 998536da7be614a24a93f9e7ec31dbca 2023-02-11 04:00:40,507 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=207 updating hbase:meta row=393f61e585ea6ccae9181b305adcbdf8, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:40,507 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088040507"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088040507"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088040507"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088040507"}]},"ts":"1676088040507"} 2023-02-11 04:00:40,508 INFO [StoreOpener-660b743875c88bf2eaa58aab9da56413-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 660b743875c88bf2eaa58aab9da56413 2023-02-11 04:00:40,508 DEBUG [StoreOpener-998536da7be614a24a93f9e7ec31dbca-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca/cf 2023-02-11 04:00:40,508 DEBUG [StoreOpener-998536da7be614a24a93f9e7ec31dbca-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca/cf 2023-02-11 04:00:40,509 INFO [StoreOpener-998536da7be614a24a93f9e7ec31dbca-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 998536da7be614a24a93f9e7ec31dbca columnFamilyName cf 2023-02-11 04:00:40,509 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=213, resume processing ppid=204 2023-02-11 04:00:40,509 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=212, resume processing ppid=206 2023-02-11 04:00:40,509 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=212, ppid=206, state=SUCCESS; OpenRegionProcedure ee86f4ea9182f5cda2013084d379724f, server=jenkins-hbase4.apache.org,40829,1676088013739 in 169 msec 2023-02-11 04:00:40,509 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=213, ppid=204, state=SUCCESS; OpenRegionProcedure a388dd856b0a069ab94fe6f6b9fb6351, server=jenkins-hbase4.apache.org,42385,1676088013771 in 168 msec 2023-02-11 04:00:40,510 INFO [StoreOpener-998536da7be614a24a93f9e7ec31dbca-1] regionserver.HStore(310): Store=998536da7be614a24a93f9e7ec31dbca/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:40,510 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=204, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=a388dd856b0a069ab94fe6f6b9fb6351, ASSIGN in 331 msec 2023-02-11 04:00:40,510 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=206, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=ee86f4ea9182f5cda2013084d379724f, ASSIGN in 331 msec 2023-02-11 04:00:40,511 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=211, resume processing ppid=207 2023-02-11 04:00:40,511 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=211, ppid=207, state=SUCCESS; OpenRegionProcedure 393f61e585ea6ccae9181b305adcbdf8, server=jenkins-hbase4.apache.org,46851,1676088013694 in 172 msec 2023-02-11 04:00:40,511 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=207, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=393f61e585ea6ccae9181b305adcbdf8, ASSIGN in 333 msec 2023-02-11 04:00:40,517 INFO [StoreOpener-73f6bdfd53354bbf7c9d40fd769c9e56-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 73f6bdfd53354bbf7c9d40fd769c9e56 2023-02-11 04:00:40,518 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca 2023-02-11 04:00:40,518 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca 2023-02-11 04:00:40,518 DEBUG [StoreOpener-660b743875c88bf2eaa58aab9da56413-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413/cf 2023-02-11 04:00:40,518 DEBUG [StoreOpener-660b743875c88bf2eaa58aab9da56413-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413/cf 2023-02-11 04:00:40,519 DEBUG [StoreOpener-73f6bdfd53354bbf7c9d40fd769c9e56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56/cf 2023-02-11 04:00:40,519 DEBUG [StoreOpener-73f6bdfd53354bbf7c9d40fd769c9e56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56/cf 2023-02-11 04:00:40,519 INFO [StoreOpener-660b743875c88bf2eaa58aab9da56413-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 660b743875c88bf2eaa58aab9da56413 columnFamilyName cf 2023-02-11 04:00:40,519 INFO [StoreOpener-73f6bdfd53354bbf7c9d40fd769c9e56-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 73f6bdfd53354bbf7c9d40fd769c9e56 columnFamilyName cf 2023-02-11 04:00:40,520 INFO [StoreOpener-73f6bdfd53354bbf7c9d40fd769c9e56-1] regionserver.HStore(310): Store=73f6bdfd53354bbf7c9d40fd769c9e56/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:40,520 INFO [StoreOpener-660b743875c88bf2eaa58aab9da56413-1] regionserver.HStore(310): Store=660b743875c88bf2eaa58aab9da56413/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:40,521 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56 2023-02-11 04:00:40,521 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413 2023-02-11 04:00:40,521 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56 2023-02-11 04:00:40,521 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 998536da7be614a24a93f9e7ec31dbca 2023-02-11 04:00:40,521 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413 2023-02-11 04:00:40,524 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 04:00:40,524 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 73f6bdfd53354bbf7c9d40fd769c9e56 2023-02-11 04:00:40,524 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 998536da7be614a24a93f9e7ec31dbca; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71273027, jitterRate=0.06205086410045624}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:40,524 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 660b743875c88bf2eaa58aab9da56413 2023-02-11 04:00:40,524 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 998536da7be614a24a93f9e7ec31dbca: Running coprocessor pre-open hook at 1676088040505Writing region info on filesystem at 1676088040505Initializing all the Stores at 1676088040506 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088040506Cleaning up temporary data from old regions at 1676088040519 (+13 ms)Cleaning up detritus from prior splits at 1676088040519Running coprocessor post-open hooks at 1676088040524 (+5 ms)Region opened successfully at 1676088040524 2023-02-11 04:00:40,525 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca., pid=210, masterSystemTime=1676088040489 2023-02-11 04:00:40,527 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 04:00:40,527 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 04:00:40,527 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 660b743875c88bf2eaa58aab9da56413; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60109528, jitterRate=-0.10429823398590088}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:40,528 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 660b743875c88bf2eaa58aab9da56413: Running coprocessor pre-open hook at 1676088040506Writing region info on filesystem at 1676088040506Initializing all the Stores at 1676088040508 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088040508Cleaning up temporary data from old regions at 1676088040521 (+13 ms)Cleaning up detritus from prior splits at 1676088040522 (+1 ms)Running coprocessor post-open hooks at 1676088040528 (+6 ms)Region opened successfully at 1676088040528 2023-02-11 04:00:40,528 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca. 2023-02-11 04:00:40,528 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca. 2023-02-11 04:00:40,528 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 73f6bdfd53354bbf7c9d40fd769c9e56; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72010723, jitterRate=0.07304339110851288}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:40,528 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 73f6bdfd53354bbf7c9d40fd769c9e56: Running coprocessor pre-open hook at 1676088040505Writing region info on filesystem at 1676088040505Initializing all the Stores at 1676088040508 (+3 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088040508Cleaning up temporary data from old regions at 1676088040521 (+13 ms)Cleaning up detritus from prior splits at 1676088040522 (+1 ms)Running coprocessor post-open hooks at 1676088040528 (+6 ms)Region opened successfully at 1676088040528 2023-02-11 04:00:40,528 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413., pid=209, masterSystemTime=1676088040487 2023-02-11 04:00:40,528 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=208 updating hbase:meta row=998536da7be614a24a93f9e7ec31dbca, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:40,528 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676088040528"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088040528"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088040528"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088040528"}]},"ts":"1676088040528"} 2023-02-11 04:00:40,529 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56., pid=214, masterSystemTime=1676088040489 2023-02-11 04:00:40,529 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413. 2023-02-11 04:00:40,530 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413. 2023-02-11 04:00:40,530 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=205 updating hbase:meta row=660b743875c88bf2eaa58aab9da56413, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:40,530 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088040530"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088040530"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088040530"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088040530"}]},"ts":"1676088040530"} 2023-02-11 04:00:40,530 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. 2023-02-11 04:00:40,530 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. 2023-02-11 04:00:40,530 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=203 updating hbase:meta row=73f6bdfd53354bbf7c9d40fd769c9e56, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:40,531 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676088040530"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088040530"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088040530"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088040530"}]},"ts":"1676088040530"} 2023-02-11 04:00:40,531 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=210, resume processing ppid=208 2023-02-11 04:00:40,531 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=210, ppid=208, state=SUCCESS; OpenRegionProcedure 998536da7be614a24a93f9e7ec31dbca, server=jenkins-hbase4.apache.org,42385,1676088013771 in 194 msec 2023-02-11 04:00:40,533 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=208, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=998536da7be614a24a93f9e7ec31dbca, ASSIGN in 353 msec 2023-02-11 04:00:40,534 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=209, resume processing ppid=205 2023-02-11 04:00:40,534 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=214, resume processing ppid=203 2023-02-11 04:00:40,534 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=209, ppid=205, state=SUCCESS; OpenRegionProcedure 660b743875c88bf2eaa58aab9da56413, server=jenkins-hbase4.apache.org,46851,1676088013694 in 196 msec 2023-02-11 04:00:40,534 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=214, ppid=203, state=SUCCESS; OpenRegionProcedure 73f6bdfd53354bbf7c9d40fd769c9e56, server=jenkins-hbase4.apache.org,40829,1676088013739 in 192 msec 2023-02-11 04:00:40,535 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=205, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=660b743875c88bf2eaa58aab9da56413, ASSIGN in 356 msec 2023-02-11 04:00:40,535 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=203, resume processing ppid=202 2023-02-11 04:00:40,535 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=203, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=73f6bdfd53354bbf7c9d40fd769c9e56, ASSIGN in 356 msec 2023-02-11 04:00:40,536 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676088039685 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-11 04:00:40,536 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676088039685","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088040536"}]},"ts":"1676088040536"} 2023-02-11 04:00:40,537 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676088039685, state=ENABLED in hbase:meta 2023-02-11 04:00:40,543 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676088039685 execute state=CREATE_TABLE_POST_OPERATION 2023-02-11 04:00:40,545 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=202, state=SUCCESS; CreateTableProcedure table=testCorruptedSnapshot-1676088039685 in 856 msec 2023-02-11 04:00:40,784 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-11 04:00:41,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-11 04:00:41,193 INFO [Listener at localhost/33753] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testCorruptedSnapshot-1676088039685, procId: 202 completed 2023-02-11 04:00:41,196 DEBUG [Listener at localhost/33753] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676088039685 2023-02-11 04:00:41,196 DEBUG [Listener at localhost/33753] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. 2023-02-11 04:00:41,199 INFO [Listener at localhost/33753] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-11 04:00:41,212 INFO [Listener at localhost/33753] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676088039685 2023-02-11 04:00:41,212 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testCorruptedSnapshot-1676088039685 2023-02-11 04:00:41,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] procedure2.ProcedureExecutor(1029): Stored pid=215, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676088039685 2023-02-11 04:00:41,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=215 2023-02-11 04:00:41,215 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676088039685","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088041215"}]},"ts":"1676088041215"} 2023-02-11 04:00:41,216 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676088039685, state=DISABLING in hbase:meta 2023-02-11 04:00:41,218 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676088039685 to state=DISABLING 2023-02-11 04:00:41,219 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=216, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=73f6bdfd53354bbf7c9d40fd769c9e56, UNASSIGN}, {pid=217, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=a388dd856b0a069ab94fe6f6b9fb6351, UNASSIGN}, {pid=218, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=660b743875c88bf2eaa58aab9da56413, UNASSIGN}, {pid=219, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=ee86f4ea9182f5cda2013084d379724f, UNASSIGN}, {pid=220, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=393f61e585ea6ccae9181b305adcbdf8, UNASSIGN}, {pid=221, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=998536da7be614a24a93f9e7ec31dbca, UNASSIGN}] 2023-02-11 04:00:41,220 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=220, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=393f61e585ea6ccae9181b305adcbdf8, UNASSIGN 2023-02-11 04:00:41,220 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=218, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=660b743875c88bf2eaa58aab9da56413, UNASSIGN 2023-02-11 04:00:41,220 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=219, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=ee86f4ea9182f5cda2013084d379724f, UNASSIGN 2023-02-11 04:00:41,220 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=221, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=998536da7be614a24a93f9e7ec31dbca, UNASSIGN 2023-02-11 04:00:41,220 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=217, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=a388dd856b0a069ab94fe6f6b9fb6351, UNASSIGN 2023-02-11 04:00:41,221 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=220 updating hbase:meta row=393f61e585ea6ccae9181b305adcbdf8, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:41,221 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=218 updating hbase:meta row=660b743875c88bf2eaa58aab9da56413, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:41,221 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=219 updating hbase:meta row=ee86f4ea9182f5cda2013084d379724f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:41,221 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088041221"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088041221"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088041221"}]},"ts":"1676088041221"} 2023-02-11 04:00:41,221 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088041221"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088041221"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088041221"}]},"ts":"1676088041221"} 2023-02-11 04:00:41,221 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088041221"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088041221"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088041221"}]},"ts":"1676088041221"} 2023-02-11 04:00:41,221 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=221 updating hbase:meta row=998536da7be614a24a93f9e7ec31dbca, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:41,221 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=217 updating hbase:meta row=a388dd856b0a069ab94fe6f6b9fb6351, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:41,221 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676088041221"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088041221"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088041221"}]},"ts":"1676088041221"} 2023-02-11 04:00:41,222 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088041221"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088041221"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088041221"}]},"ts":"1676088041221"} 2023-02-11 04:00:41,222 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=222, ppid=220, state=RUNNABLE; CloseRegionProcedure 393f61e585ea6ccae9181b305adcbdf8, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:41,223 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=223, ppid=219, state=RUNNABLE; CloseRegionProcedure ee86f4ea9182f5cda2013084d379724f, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:41,224 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=224, ppid=218, state=RUNNABLE; CloseRegionProcedure 660b743875c88bf2eaa58aab9da56413, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:41,224 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=225, ppid=221, state=RUNNABLE; CloseRegionProcedure 998536da7be614a24a93f9e7ec31dbca, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:41,225 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=226, ppid=217, state=RUNNABLE; CloseRegionProcedure a388dd856b0a069ab94fe6f6b9fb6351, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:41,226 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=216, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=73f6bdfd53354bbf7c9d40fd769c9e56, UNASSIGN 2023-02-11 04:00:41,227 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=216 updating hbase:meta row=73f6bdfd53354bbf7c9d40fd769c9e56, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:41,227 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676088041227"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088041227"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088041227"}]},"ts":"1676088041227"} 2023-02-11 04:00:41,228 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=227, ppid=216, state=RUNNABLE; CloseRegionProcedure 73f6bdfd53354bbf7c9d40fd769c9e56, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:41,375 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 393f61e585ea6ccae9181b305adcbdf8 2023-02-11 04:00:41,375 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ee86f4ea9182f5cda2013084d379724f 2023-02-11 04:00:41,375 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 393f61e585ea6ccae9181b305adcbdf8, disabling compactions & flushes 2023-02-11 04:00:41,375 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing ee86f4ea9182f5cda2013084d379724f, disabling compactions & flushes 2023-02-11 04:00:41,375 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8. 2023-02-11 04:00:41,375 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f. 2023-02-11 04:00:41,375 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8. 2023-02-11 04:00:41,375 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f. 2023-02-11 04:00:41,375 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8. after waiting 0 ms 2023-02-11 04:00:41,375 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f. after waiting 0 ms 2023-02-11 04:00:41,375 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8. 2023-02-11 04:00:41,375 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f. 2023-02-11 04:00:41,377 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 998536da7be614a24a93f9e7ec31dbca 2023-02-11 04:00:41,377 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 998536da7be614a24a93f9e7ec31dbca, disabling compactions & flushes 2023-02-11 04:00:41,377 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca. 2023-02-11 04:00:41,377 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca. 2023-02-11 04:00:41,377 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca. after waiting 0 ms 2023-02-11 04:00:41,377 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca. 2023-02-11 04:00:41,379 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 04:00:41,379 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 04:00:41,381 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8. 2023-02-11 04:00:41,381 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 393f61e585ea6ccae9181b305adcbdf8: Waiting for close lock at 1676088041375Running coprocessor pre-close hooks at 1676088041375Disabling compacts and flushes for region at 1676088041375Disabling writes for close at 1676088041375Writing region close event to WAL at 1676088041375Running coprocessor post-close hooks at 1676088041380 (+5 ms)Closed at 1676088041381 (+1 ms) 2023-02-11 04:00:41,382 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f. 2023-02-11 04:00:41,382 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for ee86f4ea9182f5cda2013084d379724f: Waiting for close lock at 1676088041375Running coprocessor pre-close hooks at 1676088041375Disabling compacts and flushes for region at 1676088041375Disabling writes for close at 1676088041375Writing region close event to WAL at 1676088041375Running coprocessor post-close hooks at 1676088041381 (+6 ms)Closed at 1676088041382 (+1 ms) 2023-02-11 04:00:41,383 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 04:00:41,383 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 393f61e585ea6ccae9181b305adcbdf8 2023-02-11 04:00:41,383 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 660b743875c88bf2eaa58aab9da56413 2023-02-11 04:00:41,383 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 660b743875c88bf2eaa58aab9da56413, disabling compactions & flushes 2023-02-11 04:00:41,384 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca. 2023-02-11 04:00:41,384 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413. 2023-02-11 04:00:41,384 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 998536da7be614a24a93f9e7ec31dbca: Waiting for close lock at 1676088041377Running coprocessor pre-close hooks at 1676088041377Disabling compacts and flushes for region at 1676088041377Disabling writes for close at 1676088041377Writing region close event to WAL at 1676088041377Running coprocessor post-close hooks at 1676088041383 (+6 ms)Closed at 1676088041384 (+1 ms) 2023-02-11 04:00:41,384 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413. 2023-02-11 04:00:41,384 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413. after waiting 0 ms 2023-02-11 04:00:41,384 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413. 2023-02-11 04:00:41,384 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=220 updating hbase:meta row=393f61e585ea6ccae9181b305adcbdf8, regionState=CLOSED 2023-02-11 04:00:41,385 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088041384"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088041384"}]},"ts":"1676088041384"} 2023-02-11 04:00:41,385 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ee86f4ea9182f5cda2013084d379724f 2023-02-11 04:00:41,385 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 73f6bdfd53354bbf7c9d40fd769c9e56 2023-02-11 04:00:41,385 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 73f6bdfd53354bbf7c9d40fd769c9e56, disabling compactions & flushes 2023-02-11 04:00:41,385 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. 2023-02-11 04:00:41,385 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. 2023-02-11 04:00:41,385 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. after waiting 0 ms 2023-02-11 04:00:41,385 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. 2023-02-11 04:00:41,386 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=219 updating hbase:meta row=ee86f4ea9182f5cda2013084d379724f, regionState=CLOSED 2023-02-11 04:00:41,386 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088041386"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088041386"}]},"ts":"1676088041386"} 2023-02-11 04:00:41,386 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 998536da7be614a24a93f9e7ec31dbca 2023-02-11 04:00:41,386 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a388dd856b0a069ab94fe6f6b9fb6351 2023-02-11 04:00:41,386 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=221 updating hbase:meta row=998536da7be614a24a93f9e7ec31dbca, regionState=CLOSED 2023-02-11 04:00:41,386 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing a388dd856b0a069ab94fe6f6b9fb6351, disabling compactions & flushes 2023-02-11 04:00:41,386 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351. 2023-02-11 04:00:41,386 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351. 2023-02-11 04:00:41,386 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351. after waiting 0 ms 2023-02-11 04:00:41,387 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351. 2023-02-11 04:00:41,386 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676088041386"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088041386"}]},"ts":"1676088041386"} 2023-02-11 04:00:41,388 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=222, resume processing ppid=220 2023-02-11 04:00:41,389 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=222, ppid=220, state=SUCCESS; CloseRegionProcedure 393f61e585ea6ccae9181b305adcbdf8, server=jenkins-hbase4.apache.org,46851,1676088013694 in 164 msec 2023-02-11 04:00:41,389 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 04:00:41,389 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=223, resume processing ppid=219 2023-02-11 04:00:41,390 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=223, ppid=219, state=SUCCESS; CloseRegionProcedure ee86f4ea9182f5cda2013084d379724f, server=jenkins-hbase4.apache.org,40829,1676088013739 in 164 msec 2023-02-11 04:00:41,391 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. 2023-02-11 04:00:41,391 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 73f6bdfd53354bbf7c9d40fd769c9e56: Waiting for close lock at 1676088041385Running coprocessor pre-close hooks at 1676088041385Disabling compacts and flushes for region at 1676088041385Disabling writes for close at 1676088041385Writing region close event to WAL at 1676088041386 (+1 ms)Running coprocessor post-close hooks at 1676088041390 (+4 ms)Closed at 1676088041391 (+1 ms) 2023-02-11 04:00:41,391 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 04:00:41,392 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351. 2023-02-11 04:00:41,392 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for a388dd856b0a069ab94fe6f6b9fb6351: Waiting for close lock at 1676088041386Running coprocessor pre-close hooks at 1676088041386Disabling compacts and flushes for region at 1676088041386Disabling writes for close at 1676088041387 (+1 ms)Writing region close event to WAL at 1676088041387Running coprocessor post-close hooks at 1676088041392 (+5 ms)Closed at 1676088041392 2023-02-11 04:00:41,393 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 04:00:41,394 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413. 2023-02-11 04:00:41,394 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 660b743875c88bf2eaa58aab9da56413: Waiting for close lock at 1676088041383Running coprocessor pre-close hooks at 1676088041383Disabling compacts and flushes for region at 1676088041383Disabling writes for close at 1676088041384 (+1 ms)Writing region close event to WAL at 1676088041384Running coprocessor post-close hooks at 1676088041394 (+10 ms)Closed at 1676088041394 2023-02-11 04:00:41,395 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=220, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=393f61e585ea6ccae9181b305adcbdf8, UNASSIGN in 170 msec 2023-02-11 04:00:41,396 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 73f6bdfd53354bbf7c9d40fd769c9e56 2023-02-11 04:00:41,396 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=225, resume processing ppid=221 2023-02-11 04:00:41,396 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=219, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=ee86f4ea9182f5cda2013084d379724f, UNASSIGN in 171 msec 2023-02-11 04:00:41,396 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=225, ppid=221, state=SUCCESS; CloseRegionProcedure 998536da7be614a24a93f9e7ec31dbca, server=jenkins-hbase4.apache.org,42385,1676088013771 in 164 msec 2023-02-11 04:00:41,400 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=216 updating hbase:meta row=73f6bdfd53354bbf7c9d40fd769c9e56, regionState=CLOSED 2023-02-11 04:00:41,400 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676088041400"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088041400"}]},"ts":"1676088041400"} 2023-02-11 04:00:41,400 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a388dd856b0a069ab94fe6f6b9fb6351 2023-02-11 04:00:41,401 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=217 updating hbase:meta row=a388dd856b0a069ab94fe6f6b9fb6351, regionState=CLOSED 2023-02-11 04:00:41,401 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=221, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=998536da7be614a24a93f9e7ec31dbca, UNASSIGN in 177 msec 2023-02-11 04:00:41,401 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088041401"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088041401"}]},"ts":"1676088041401"} 2023-02-11 04:00:41,401 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 660b743875c88bf2eaa58aab9da56413 2023-02-11 04:00:41,401 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=218 updating hbase:meta row=660b743875c88bf2eaa58aab9da56413, regionState=CLOSED 2023-02-11 04:00:41,401 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088041401"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088041401"}]},"ts":"1676088041401"} 2023-02-11 04:00:41,403 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=227, resume processing ppid=216 2023-02-11 04:00:41,403 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=227, ppid=216, state=SUCCESS; CloseRegionProcedure 73f6bdfd53354bbf7c9d40fd769c9e56, server=jenkins-hbase4.apache.org,40829,1676088013739 in 173 msec 2023-02-11 04:00:41,404 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=226, resume processing ppid=217 2023-02-11 04:00:41,404 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=226, ppid=217, state=SUCCESS; CloseRegionProcedure a388dd856b0a069ab94fe6f6b9fb6351, server=jenkins-hbase4.apache.org,42385,1676088013771 in 177 msec 2023-02-11 04:00:41,404 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=216, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=73f6bdfd53354bbf7c9d40fd769c9e56, UNASSIGN in 184 msec 2023-02-11 04:00:41,405 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=224, resume processing ppid=218 2023-02-11 04:00:41,405 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=224, ppid=218, state=SUCCESS; CloseRegionProcedure 660b743875c88bf2eaa58aab9da56413, server=jenkins-hbase4.apache.org,46851,1676088013694 in 179 msec 2023-02-11 04:00:41,405 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=217, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=a388dd856b0a069ab94fe6f6b9fb6351, UNASSIGN in 185 msec 2023-02-11 04:00:41,406 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=218, resume processing ppid=215 2023-02-11 04:00:41,406 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=218, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=660b743875c88bf2eaa58aab9da56413, UNASSIGN in 186 msec 2023-02-11 04:00:41,407 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676088039685","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088041407"}]},"ts":"1676088041407"} 2023-02-11 04:00:41,408 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676088039685, state=DISABLED in hbase:meta 2023-02-11 04:00:41,411 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676088039685 to state=DISABLED 2023-02-11 04:00:41,413 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=215, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676088039685 in 200 msec 2023-02-11 04:00:41,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=215 2023-02-11 04:00:41,466 INFO [Listener at localhost/33753] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676088039685, procId: 215 completed 2023-02-11 04:00:41,467 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=emptySnaptb-1676088039685 table=testCorruptedSnapshot-1676088039685 type=FLUSH ttl=0 } 2023-02-11 04:00:41,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676088041467 (current time:1676088041467). 2023-02-11 04:00:41,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-11 04:00:41,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] zookeeper.ReadOnlyZKClient(139): Connect 0x72f16bfd to 127.0.0.1:63425 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 04:00:41,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7a697be0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 04:00:41,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 04:00:41,475 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39100, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 04:00:41,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x72f16bfd to 127.0.0.1:63425 2023-02-11 04:00:41,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:00:41,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-11 04:00:41,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=emptySnaptb-1676088039685 table=testCorruptedSnapshot-1676088039685 type=FLUSH ttl=0 } 2023-02-11 04:00:41,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] procedure2.ProcedureExecutor(1029): Stored pid=228, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676088039685, type=EXCLUSIVE 2023-02-11 04:00:41,479 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=228, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676088039685, type=EXCLUSIVE 2023-02-11 04:00:41,480 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=228, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676088039685, type=EXCLUSIVE; timeout=600000, timestamp=1676088641480 2023-02-11 04:00:41,480 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot emptySnaptb-1676088039685 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676088039685 2023-02-11 04:00:41,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(664): Started snapshot: { ss=emptySnaptb-1676088039685 table=testCorruptedSnapshot-1676088039685 type=FLUSH ttl=0 } 2023-02-11 04:00:41,480 DEBUG [Listener at localhost/33753] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=emptySnaptb-1676088039685 table=testCorruptedSnapshot-1676088039685 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-11 04:00:41,480 DEBUG [Listener at localhost/33753] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-11 04:00:41,493 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676088039685 table=testCorruptedSnapshot-1676088039685 type=DISABLED ttl=0 } 2023-02-11 04:00:41,499 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:41,500 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:41,500 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:41,500 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:41,500 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:41,500 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:41,500 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:41,500 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:41,501 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:41,501 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:41,501 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:41,501 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:41,502 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 04:00:41,502 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 04:00:41,508 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:41,508 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:41,508 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 04:00:41,509 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 04:00:41,509 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 04:00:41,510 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 04:00:41,510 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 04:00:41,567 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=emptySnaptb-1676088039685 table=testCorruptedSnapshot-1676088039685 type=DISABLED ttl=0 } as finished. 2023-02-11 04:00:41,567 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for emptySnaptb-1676088039685 2023-02-11 04:00:41,567 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/.tmp/emptySnaptb-1676088039685 2023-02-11 04:00:41,591 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/.tmp/emptySnaptb-1676088039685 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/emptySnaptb-1676088039685 2023-02-11 04:00:41,611 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot emptySnaptb-1676088039685 of table testCorruptedSnapshot-1676088039685 completed 2023-02-11 04:00:41,611 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/.tmp/emptySnaptb-1676088039685 2023-02-11 04:00:41,611 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/.tmp/emptySnaptb-1676088039685 2023-02-11 04:00:41,611 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot emptySnaptb-1676088039685 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676088039685 at 1676088041480Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676088039685 table=testCorruptedSnapshot-1676088039685 type=DISABLED ttl=0 } at 1676088041493 (+13 ms)Consolidate snapshot: emptySnaptb-1676088039685 at 1676088041567 (+74 ms)Loading Region manifests for emptySnaptb-1676088039685 at 1676088041567Writing data manifest for emptySnaptb-1676088039685 at 1676088041571 (+4 ms)Verifying snapshot: emptySnaptb-1676088039685 at 1676088041583 (+12 ms)Snapshot emptySnaptb-1676088039685 of table testCorruptedSnapshot-1676088039685 completed at 1676088041611 (+28 ms) 2023-02-11 04:00:41,613 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=228, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676088039685, type=EXCLUSIVE 2023-02-11 04:00:41,614 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=228, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676088039685, type=EXCLUSIVE in 136 msec 2023-02-11 04:00:41,730 DEBUG [Listener at localhost/33753] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 04:00:41,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676088039685 table=testCorruptedSnapshot-1676088039685 type=FLUSH ttl=0 } is done 2023-02-11 04:00:41,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(401): Snapshot '{ ss=emptySnaptb-1676088039685 table=testCorruptedSnapshot-1676088039685 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-11 04:00:41,732 INFO [Listener at localhost/33753] client.HBaseAdmin$14(830): Started enable of testCorruptedSnapshot-1676088039685 2023-02-11 04:00:41,732 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testCorruptedSnapshot-1676088039685 2023-02-11 04:00:41,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] procedure2.ProcedureExecutor(1029): Stored pid=229, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testCorruptedSnapshot-1676088039685 2023-02-11 04:00:41,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-11 04:00:41,735 INFO [PEWorker-4] procedure.EnableTableProcedure(296): Attempting to enable the table testCorruptedSnapshot-1676088039685 2023-02-11 04:00:41,735 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676088039685","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088041735"}]},"ts":"1676088041735"} 2023-02-11 04:00:41,736 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676088039685, state=ENABLING in hbase:meta 2023-02-11 04:00:41,738 DEBUG [PEWorker-4] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-11 04:00:41,739 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=73f6bdfd53354bbf7c9d40fd769c9e56, ASSIGN}, {pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=a388dd856b0a069ab94fe6f6b9fb6351, ASSIGN}, {pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=660b743875c88bf2eaa58aab9da56413, ASSIGN}, {pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=ee86f4ea9182f5cda2013084d379724f, ASSIGN}, {pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=393f61e585ea6ccae9181b305adcbdf8, ASSIGN}, {pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=998536da7be614a24a93f9e7ec31dbca, ASSIGN}] 2023-02-11 04:00:41,741 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=a388dd856b0a069ab94fe6f6b9fb6351, ASSIGN 2023-02-11 04:00:41,741 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=ee86f4ea9182f5cda2013084d379724f, ASSIGN 2023-02-11 04:00:41,741 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=73f6bdfd53354bbf7c9d40fd769c9e56, ASSIGN 2023-02-11 04:00:41,742 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=660b743875c88bf2eaa58aab9da56413, ASSIGN 2023-02-11 04:00:41,742 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=998536da7be614a24a93f9e7ec31dbca, ASSIGN 2023-02-11 04:00:41,742 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:41,742 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:41,742 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=a388dd856b0a069ab94fe6f6b9fb6351, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,42385,1676088013771; forceNewPlan=false, retain=true 2023-02-11 04:00:41,742 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=660b743875c88bf2eaa58aab9da56413, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,46851,1676088013694; forceNewPlan=false, retain=true 2023-02-11 04:00:41,742 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:41,743 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:41,743 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=998536da7be614a24a93f9e7ec31dbca, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,42385,1676088013771; forceNewPlan=false, retain=true 2023-02-11 04:00:41,743 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:41,743 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=73f6bdfd53354bbf7c9d40fd769c9e56, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,40829,1676088013739; forceNewPlan=false, retain=true 2023-02-11 04:00:41,743 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=ee86f4ea9182f5cda2013084d379724f, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,40829,1676088013739; forceNewPlan=false, retain=true 2023-02-11 04:00:41,743 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=393f61e585ea6ccae9181b305adcbdf8, ASSIGN 2023-02-11 04:00:41,743 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:41,743 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=393f61e585ea6ccae9181b305adcbdf8, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,46851,1676088013694; forceNewPlan=false, retain=true 2023-02-11 04:00:41,893 INFO [jenkins-hbase4:41713] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-11 04:00:41,893 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=234 updating hbase:meta row=393f61e585ea6ccae9181b305adcbdf8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:41,893 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=233 updating hbase:meta row=ee86f4ea9182f5cda2013084d379724f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:41,893 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=231 updating hbase:meta row=a388dd856b0a069ab94fe6f6b9fb6351, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:41,893 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088041893"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088041893"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088041893"}]},"ts":"1676088041893"} 2023-02-11 04:00:41,893 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088041893"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088041893"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088041893"}]},"ts":"1676088041893"} 2023-02-11 04:00:41,893 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=235 updating hbase:meta row=998536da7be614a24a93f9e7ec31dbca, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:41,893 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=232 updating hbase:meta row=660b743875c88bf2eaa58aab9da56413, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:41,893 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676088041893"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088041893"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088041893"}]},"ts":"1676088041893"} 2023-02-11 04:00:41,893 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088041893"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088041893"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088041893"}]},"ts":"1676088041893"} 2023-02-11 04:00:41,893 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088041893"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088041893"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088041893"}]},"ts":"1676088041893"} 2023-02-11 04:00:41,894 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=236, ppid=234, state=RUNNABLE; OpenRegionProcedure 393f61e585ea6ccae9181b305adcbdf8, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:41,895 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=237, ppid=231, state=RUNNABLE; OpenRegionProcedure a388dd856b0a069ab94fe6f6b9fb6351, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:41,895 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=238, ppid=235, state=RUNNABLE; OpenRegionProcedure 998536da7be614a24a93f9e7ec31dbca, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:41,896 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=239, ppid=232, state=RUNNABLE; OpenRegionProcedure 660b743875c88bf2eaa58aab9da56413, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:41,897 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=240, ppid=233, state=RUNNABLE; OpenRegionProcedure ee86f4ea9182f5cda2013084d379724f, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:41,898 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=230 updating hbase:meta row=73f6bdfd53354bbf7c9d40fd769c9e56, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:41,898 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676088041898"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088041898"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088041898"}]},"ts":"1676088041898"} 2023-02-11 04:00:41,899 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=241, ppid=230, state=RUNNABLE; OpenRegionProcedure 73f6bdfd53354bbf7c9d40fd769c9e56, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:41,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-11 04:00:42,049 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8. 2023-02-11 04:00:42,049 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca. 2023-02-11 04:00:42,049 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 393f61e585ea6ccae9181b305adcbdf8, NAME => 'testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 04:00:42,049 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 998536da7be614a24a93f9e7ec31dbca, NAME => 'testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 04:00:42,049 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f. 2023-02-11 04:00:42,050 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676088039685 393f61e585ea6ccae9181b305adcbdf8 2023-02-11 04:00:42,050 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ee86f4ea9182f5cda2013084d379724f, NAME => 'testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 04:00:42,050 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:42,050 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676088039685 998536da7be614a24a93f9e7ec31dbca 2023-02-11 04:00:42,050 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 393f61e585ea6ccae9181b305adcbdf8 2023-02-11 04:00:42,050 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:42,050 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 393f61e585ea6ccae9181b305adcbdf8 2023-02-11 04:00:42,050 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676088039685 ee86f4ea9182f5cda2013084d379724f 2023-02-11 04:00:42,050 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 998536da7be614a24a93f9e7ec31dbca 2023-02-11 04:00:42,050 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:42,050 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 998536da7be614a24a93f9e7ec31dbca 2023-02-11 04:00:42,050 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for ee86f4ea9182f5cda2013084d379724f 2023-02-11 04:00:42,050 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for ee86f4ea9182f5cda2013084d379724f 2023-02-11 04:00:42,051 INFO [StoreOpener-393f61e585ea6ccae9181b305adcbdf8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 393f61e585ea6ccae9181b305adcbdf8 2023-02-11 04:00:42,051 INFO [StoreOpener-998536da7be614a24a93f9e7ec31dbca-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 998536da7be614a24a93f9e7ec31dbca 2023-02-11 04:00:42,051 INFO [StoreOpener-ee86f4ea9182f5cda2013084d379724f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ee86f4ea9182f5cda2013084d379724f 2023-02-11 04:00:42,052 DEBUG [StoreOpener-393f61e585ea6ccae9181b305adcbdf8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8/cf 2023-02-11 04:00:42,052 DEBUG [StoreOpener-393f61e585ea6ccae9181b305adcbdf8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8/cf 2023-02-11 04:00:42,052 DEBUG [StoreOpener-998536da7be614a24a93f9e7ec31dbca-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca/cf 2023-02-11 04:00:42,052 DEBUG [StoreOpener-998536da7be614a24a93f9e7ec31dbca-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca/cf 2023-02-11 04:00:42,052 DEBUG [StoreOpener-ee86f4ea9182f5cda2013084d379724f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f/cf 2023-02-11 04:00:42,052 DEBUG [StoreOpener-ee86f4ea9182f5cda2013084d379724f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f/cf 2023-02-11 04:00:42,052 INFO [StoreOpener-393f61e585ea6ccae9181b305adcbdf8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 393f61e585ea6ccae9181b305adcbdf8 columnFamilyName cf 2023-02-11 04:00:42,052 INFO [StoreOpener-ee86f4ea9182f5cda2013084d379724f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ee86f4ea9182f5cda2013084d379724f columnFamilyName cf 2023-02-11 04:00:42,052 INFO [StoreOpener-998536da7be614a24a93f9e7ec31dbca-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 998536da7be614a24a93f9e7ec31dbca columnFamilyName cf 2023-02-11 04:00:42,053 INFO [StoreOpener-393f61e585ea6ccae9181b305adcbdf8-1] regionserver.HStore(310): Store=393f61e585ea6ccae9181b305adcbdf8/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:42,053 INFO [StoreOpener-ee86f4ea9182f5cda2013084d379724f-1] regionserver.HStore(310): Store=ee86f4ea9182f5cda2013084d379724f/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:42,053 INFO [StoreOpener-998536da7be614a24a93f9e7ec31dbca-1] regionserver.HStore(310): Store=998536da7be614a24a93f9e7ec31dbca/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:42,054 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8 2023-02-11 04:00:42,054 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f 2023-02-11 04:00:42,054 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca 2023-02-11 04:00:42,054 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8 2023-02-11 04:00:42,054 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f 2023-02-11 04:00:42,055 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca 2023-02-11 04:00:42,057 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for ee86f4ea9182f5cda2013084d379724f 2023-02-11 04:00:42,057 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 393f61e585ea6ccae9181b305adcbdf8 2023-02-11 04:00:42,057 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 998536da7be614a24a93f9e7ec31dbca 2023-02-11 04:00:42,057 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened ee86f4ea9182f5cda2013084d379724f; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62482073, jitterRate=-0.06894455850124359}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:42,057 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for ee86f4ea9182f5cda2013084d379724f: Running coprocessor pre-open hook at 1676088042050Writing region info on filesystem at 1676088042050Initializing all the Stores at 1676088042051 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088042051Cleaning up temporary data from old regions at 1676088042055 (+4 ms)Cleaning up detritus from prior splits at 1676088042055Running coprocessor post-open hooks at 1676088042057 (+2 ms)Region opened successfully at 1676088042057 2023-02-11 04:00:42,057 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 393f61e585ea6ccae9181b305adcbdf8; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71175036, jitterRate=0.06059068441390991}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:42,057 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 393f61e585ea6ccae9181b305adcbdf8: Running coprocessor pre-open hook at 1676088042050Writing region info on filesystem at 1676088042050Initializing all the Stores at 1676088042051 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088042051Cleaning up temporary data from old regions at 1676088042055 (+4 ms)Cleaning up detritus from prior splits at 1676088042055Running coprocessor post-open hooks at 1676088042057 (+2 ms)Region opened successfully at 1676088042057 2023-02-11 04:00:42,058 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 998536da7be614a24a93f9e7ec31dbca; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58766235, jitterRate=-0.12431485950946808}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:42,058 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 998536da7be614a24a93f9e7ec31dbca: Running coprocessor pre-open hook at 1676088042050Writing region info on filesystem at 1676088042050Initializing all the Stores at 1676088042051 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088042051Cleaning up temporary data from old regions at 1676088042055 (+4 ms)Cleaning up detritus from prior splits at 1676088042056 (+1 ms)Running coprocessor post-open hooks at 1676088042058 (+2 ms)Region opened successfully at 1676088042058 2023-02-11 04:00:42,058 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f., pid=240, masterSystemTime=1676088042048 2023-02-11 04:00:42,058 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8., pid=236, masterSystemTime=1676088042047 2023-02-11 04:00:42,058 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca., pid=238, masterSystemTime=1676088042047 2023-02-11 04:00:42,059 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f. 2023-02-11 04:00:42,059 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f. 2023-02-11 04:00:42,059 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. 2023-02-11 04:00:42,060 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 73f6bdfd53354bbf7c9d40fd769c9e56, NAME => 'testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 04:00:42,060 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=233 updating hbase:meta row=ee86f4ea9182f5cda2013084d379724f, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:42,060 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088042060"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088042060"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088042060"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088042060"}]},"ts":"1676088042060"} 2023-02-11 04:00:42,060 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676088039685 73f6bdfd53354bbf7c9d40fd769c9e56 2023-02-11 04:00:42,060 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:42,060 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8. 2023-02-11 04:00:42,060 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 73f6bdfd53354bbf7c9d40fd769c9e56 2023-02-11 04:00:42,060 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8. 2023-02-11 04:00:42,060 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 73f6bdfd53354bbf7c9d40fd769c9e56 2023-02-11 04:00:42,060 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413. 2023-02-11 04:00:42,060 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 660b743875c88bf2eaa58aab9da56413, NAME => 'testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 04:00:42,061 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=234 updating hbase:meta row=393f61e585ea6ccae9181b305adcbdf8, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:42,061 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676088039685 660b743875c88bf2eaa58aab9da56413 2023-02-11 04:00:42,061 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088042061"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088042061"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088042061"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088042061"}]},"ts":"1676088042061"} 2023-02-11 04:00:42,061 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:42,061 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 660b743875c88bf2eaa58aab9da56413 2023-02-11 04:00:42,061 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 660b743875c88bf2eaa58aab9da56413 2023-02-11 04:00:42,061 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca. 2023-02-11 04:00:42,061 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca. 2023-02-11 04:00:42,061 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351. 2023-02-11 04:00:42,061 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a388dd856b0a069ab94fe6f6b9fb6351, NAME => 'testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 04:00:42,061 INFO [StoreOpener-73f6bdfd53354bbf7c9d40fd769c9e56-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 73f6bdfd53354bbf7c9d40fd769c9e56 2023-02-11 04:00:42,062 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676088039685 a388dd856b0a069ab94fe6f6b9fb6351 2023-02-11 04:00:42,062 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:42,062 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for a388dd856b0a069ab94fe6f6b9fb6351 2023-02-11 04:00:42,062 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for a388dd856b0a069ab94fe6f6b9fb6351 2023-02-11 04:00:42,062 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=235 updating hbase:meta row=998536da7be614a24a93f9e7ec31dbca, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:42,062 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676088042062"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088042062"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088042062"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088042062"}]},"ts":"1676088042062"} 2023-02-11 04:00:42,064 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=240, resume processing ppid=233 2023-02-11 04:00:42,064 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=236, resume processing ppid=234 2023-02-11 04:00:42,064 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=240, ppid=233, state=SUCCESS; OpenRegionProcedure ee86f4ea9182f5cda2013084d379724f, server=jenkins-hbase4.apache.org,40829,1676088013739 in 166 msec 2023-02-11 04:00:42,064 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=236, ppid=234, state=SUCCESS; OpenRegionProcedure 393f61e585ea6ccae9181b305adcbdf8, server=jenkins-hbase4.apache.org,46851,1676088013694 in 169 msec 2023-02-11 04:00:42,065 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=238, resume processing ppid=235 2023-02-11 04:00:42,065 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=238, ppid=235, state=SUCCESS; OpenRegionProcedure 998536da7be614a24a93f9e7ec31dbca, server=jenkins-hbase4.apache.org,42385,1676088013771 in 168 msec 2023-02-11 04:00:42,065 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=233, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=ee86f4ea9182f5cda2013084d379724f, ASSIGN in 325 msec 2023-02-11 04:00:42,065 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=234, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=393f61e585ea6ccae9181b305adcbdf8, ASSIGN in 325 msec 2023-02-11 04:00:42,066 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=235, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=998536da7be614a24a93f9e7ec31dbca, ASSIGN in 326 msec 2023-02-11 04:00:42,068 DEBUG [StoreOpener-73f6bdfd53354bbf7c9d40fd769c9e56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56/cf 2023-02-11 04:00:42,068 DEBUG [StoreOpener-73f6bdfd53354bbf7c9d40fd769c9e56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56/cf 2023-02-11 04:00:42,068 INFO [StoreOpener-660b743875c88bf2eaa58aab9da56413-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 660b743875c88bf2eaa58aab9da56413 2023-02-11 04:00:42,068 INFO [StoreOpener-a388dd856b0a069ab94fe6f6b9fb6351-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a388dd856b0a069ab94fe6f6b9fb6351 2023-02-11 04:00:42,069 INFO [StoreOpener-73f6bdfd53354bbf7c9d40fd769c9e56-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 73f6bdfd53354bbf7c9d40fd769c9e56 columnFamilyName cf 2023-02-11 04:00:42,069 DEBUG [StoreOpener-660b743875c88bf2eaa58aab9da56413-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413/cf 2023-02-11 04:00:42,069 DEBUG [StoreOpener-660b743875c88bf2eaa58aab9da56413-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413/cf 2023-02-11 04:00:42,069 DEBUG [StoreOpener-a388dd856b0a069ab94fe6f6b9fb6351-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351/cf 2023-02-11 04:00:42,069 INFO [StoreOpener-73f6bdfd53354bbf7c9d40fd769c9e56-1] regionserver.HStore(310): Store=73f6bdfd53354bbf7c9d40fd769c9e56/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:42,069 DEBUG [StoreOpener-a388dd856b0a069ab94fe6f6b9fb6351-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351/cf 2023-02-11 04:00:42,069 INFO [StoreOpener-660b743875c88bf2eaa58aab9da56413-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 660b743875c88bf2eaa58aab9da56413 columnFamilyName cf 2023-02-11 04:00:42,070 INFO [StoreOpener-a388dd856b0a069ab94fe6f6b9fb6351-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a388dd856b0a069ab94fe6f6b9fb6351 columnFamilyName cf 2023-02-11 04:00:42,070 INFO [StoreOpener-660b743875c88bf2eaa58aab9da56413-1] regionserver.HStore(310): Store=660b743875c88bf2eaa58aab9da56413/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:42,070 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56 2023-02-11 04:00:42,071 INFO [StoreOpener-a388dd856b0a069ab94fe6f6b9fb6351-1] regionserver.HStore(310): Store=a388dd856b0a069ab94fe6f6b9fb6351/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:42,071 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413 2023-02-11 04:00:42,071 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351 2023-02-11 04:00:42,071 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56 2023-02-11 04:00:42,072 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413 2023-02-11 04:00:42,072 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351 2023-02-11 04:00:42,073 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 73f6bdfd53354bbf7c9d40fd769c9e56 2023-02-11 04:00:42,074 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 73f6bdfd53354bbf7c9d40fd769c9e56; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68270599, jitterRate=0.017311200499534607}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:42,074 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 73f6bdfd53354bbf7c9d40fd769c9e56: Running coprocessor pre-open hook at 1676088042060Writing region info on filesystem at 1676088042060Initializing all the Stores at 1676088042061 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088042061Cleaning up temporary data from old regions at 1676088042072 (+11 ms)Cleaning up detritus from prior splits at 1676088042072Running coprocessor post-open hooks at 1676088042074 (+2 ms)Region opened successfully at 1676088042074 2023-02-11 04:00:42,075 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 660b743875c88bf2eaa58aab9da56413 2023-02-11 04:00:42,075 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56., pid=241, masterSystemTime=1676088042048 2023-02-11 04:00:42,075 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for a388dd856b0a069ab94fe6f6b9fb6351 2023-02-11 04:00:42,076 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 660b743875c88bf2eaa58aab9da56413; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68383121, jitterRate=0.018987908959388733}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:42,076 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 660b743875c88bf2eaa58aab9da56413: Running coprocessor pre-open hook at 1676088042061Writing region info on filesystem at 1676088042061Initializing all the Stores at 1676088042068 (+7 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088042068Cleaning up temporary data from old regions at 1676088042072 (+4 ms)Cleaning up detritus from prior splits at 1676088042073 (+1 ms)Running coprocessor post-open hooks at 1676088042076 (+3 ms)Region opened successfully at 1676088042076 2023-02-11 04:00:42,076 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened a388dd856b0a069ab94fe6f6b9fb6351; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67199404, jitterRate=0.0013491511344909668}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:42,076 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for a388dd856b0a069ab94fe6f6b9fb6351: Running coprocessor pre-open hook at 1676088042062Writing region info on filesystem at 1676088042062Initializing all the Stores at 1676088042068 (+6 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088042068Cleaning up temporary data from old regions at 1676088042073 (+5 ms)Cleaning up detritus from prior splits at 1676088042073Running coprocessor post-open hooks at 1676088042076 (+3 ms)Region opened successfully at 1676088042076 2023-02-11 04:00:42,076 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. 2023-02-11 04:00:42,076 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. 2023-02-11 04:00:42,076 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413., pid=239, masterSystemTime=1676088042047 2023-02-11 04:00:42,077 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351., pid=237, masterSystemTime=1676088042047 2023-02-11 04:00:42,077 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=230 updating hbase:meta row=73f6bdfd53354bbf7c9d40fd769c9e56, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:42,077 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676088042077"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088042077"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088042077"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088042077"}]},"ts":"1676088042077"} 2023-02-11 04:00:42,078 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413. 2023-02-11 04:00:42,078 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413. 2023-02-11 04:00:42,078 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=232 updating hbase:meta row=660b743875c88bf2eaa58aab9da56413, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:42,079 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088042078"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088042078"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088042078"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088042078"}]},"ts":"1676088042078"} 2023-02-11 04:00:42,079 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351. 2023-02-11 04:00:42,079 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351. 2023-02-11 04:00:42,080 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=231 updating hbase:meta row=a388dd856b0a069ab94fe6f6b9fb6351, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:42,080 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088042080"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088042080"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088042080"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088042080"}]},"ts":"1676088042080"} 2023-02-11 04:00:42,082 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=241, resume processing ppid=230 2023-02-11 04:00:42,082 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=241, ppid=230, state=SUCCESS; OpenRegionProcedure 73f6bdfd53354bbf7c9d40fd769c9e56, server=jenkins-hbase4.apache.org,40829,1676088013739 in 180 msec 2023-02-11 04:00:42,083 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=239, resume processing ppid=232 2023-02-11 04:00:42,083 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=239, ppid=232, state=SUCCESS; OpenRegionProcedure 660b743875c88bf2eaa58aab9da56413, server=jenkins-hbase4.apache.org,46851,1676088013694 in 184 msec 2023-02-11 04:00:42,084 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=230, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=73f6bdfd53354bbf7c9d40fd769c9e56, ASSIGN in 343 msec 2023-02-11 04:00:42,084 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=237, resume processing ppid=231 2023-02-11 04:00:42,084 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=237, ppid=231, state=SUCCESS; OpenRegionProcedure a388dd856b0a069ab94fe6f6b9fb6351, server=jenkins-hbase4.apache.org,42385,1676088013771 in 187 msec 2023-02-11 04:00:42,085 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=232, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=660b743875c88bf2eaa58aab9da56413, ASSIGN in 344 msec 2023-02-11 04:00:42,086 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=231, resume processing ppid=229 2023-02-11 04:00:42,086 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=231, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=a388dd856b0a069ab94fe6f6b9fb6351, ASSIGN in 345 msec 2023-02-11 04:00:42,087 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676088039685","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088042086"}]},"ts":"1676088042086"} 2023-02-11 04:00:42,088 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676088039685, state=ENABLED in hbase:meta 2023-02-11 04:00:42,092 INFO [PEWorker-3] procedure.EnableTableProcedure(310): Table 'testCorruptedSnapshot-1676088039685' was successfully enabled. 2023-02-11 04:00:42,093 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=229, state=SUCCESS; EnableTableProcedure table=testCorruptedSnapshot-1676088039685 in 360 msec 2023-02-11 04:00:42,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-11 04:00:42,486 INFO [Listener at localhost/33753] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testCorruptedSnapshot-1676088039685, procId: 229 completed 2023-02-11 04:00:42,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 04:00:42,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 04:00:42,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 04:00:42,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 04:00:42,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 04:00:42,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 04:00:42,501 DEBUG [Listener at localhost/33753] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676088039685 2023-02-11 04:00:42,501 DEBUG [Listener at localhost/33753] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. 2023-02-11 04:00:42,501 INFO [Listener at localhost/33753] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-11 04:00:42,513 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 04:00:42,514 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 04:00:42,515 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 04:00:42,516 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 04:00:42,517 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 04:00:42,520 INFO [Listener at localhost/33753] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676088039685 2023-02-11 04:00:42,520 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testCorruptedSnapshot-1676088039685 2023-02-11 04:00:42,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] procedure2.ProcedureExecutor(1029): Stored pid=242, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676088039685 2023-02-11 04:00:42,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-11 04:00:42,523 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676088039685","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088042523"}]},"ts":"1676088042523"} 2023-02-11 04:00:42,525 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676088039685, state=DISABLING in hbase:meta 2023-02-11 04:00:42,526 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676088039685 to state=DISABLING 2023-02-11 04:00:42,527 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=243, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=73f6bdfd53354bbf7c9d40fd769c9e56, UNASSIGN}, {pid=244, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=a388dd856b0a069ab94fe6f6b9fb6351, UNASSIGN}, {pid=245, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=660b743875c88bf2eaa58aab9da56413, UNASSIGN}, {pid=246, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=ee86f4ea9182f5cda2013084d379724f, UNASSIGN}, {pid=247, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=393f61e585ea6ccae9181b305adcbdf8, UNASSIGN}, {pid=248, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=998536da7be614a24a93f9e7ec31dbca, UNASSIGN}] 2023-02-11 04:00:42,527 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=246, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=ee86f4ea9182f5cda2013084d379724f, UNASSIGN 2023-02-11 04:00:42,528 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=245, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=660b743875c88bf2eaa58aab9da56413, UNASSIGN 2023-02-11 04:00:42,528 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=244, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=a388dd856b0a069ab94fe6f6b9fb6351, UNASSIGN 2023-02-11 04:00:42,528 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=243, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=73f6bdfd53354bbf7c9d40fd769c9e56, UNASSIGN 2023-02-11 04:00:42,528 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=248, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=998536da7be614a24a93f9e7ec31dbca, UNASSIGN 2023-02-11 04:00:42,528 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=246 updating hbase:meta row=ee86f4ea9182f5cda2013084d379724f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:42,528 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088042528"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088042528"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088042528"}]},"ts":"1676088042528"} 2023-02-11 04:00:42,528 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=245 updating hbase:meta row=660b743875c88bf2eaa58aab9da56413, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:42,529 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088042528"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088042528"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088042528"}]},"ts":"1676088042528"} 2023-02-11 04:00:42,529 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=244 updating hbase:meta row=a388dd856b0a069ab94fe6f6b9fb6351, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:42,529 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=243 updating hbase:meta row=73f6bdfd53354bbf7c9d40fd769c9e56, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:42,529 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=248 updating hbase:meta row=998536da7be614a24a93f9e7ec31dbca, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:42,529 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676088042529"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088042529"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088042529"}]},"ts":"1676088042529"} 2023-02-11 04:00:42,529 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088042529"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088042529"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088042529"}]},"ts":"1676088042529"} 2023-02-11 04:00:42,529 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676088042529"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088042529"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088042529"}]},"ts":"1676088042529"} 2023-02-11 04:00:42,530 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=249, ppid=246, state=RUNNABLE; CloseRegionProcedure ee86f4ea9182f5cda2013084d379724f, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:42,530 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=250, ppid=245, state=RUNNABLE; CloseRegionProcedure 660b743875c88bf2eaa58aab9da56413, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:42,531 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=251, ppid=248, state=RUNNABLE; CloseRegionProcedure 998536da7be614a24a93f9e7ec31dbca, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:42,532 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=252, ppid=244, state=RUNNABLE; CloseRegionProcedure a388dd856b0a069ab94fe6f6b9fb6351, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:42,533 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=253, ppid=243, state=RUNNABLE; CloseRegionProcedure 73f6bdfd53354bbf7c9d40fd769c9e56, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:42,533 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=247, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=393f61e585ea6ccae9181b305adcbdf8, UNASSIGN 2023-02-11 04:00:42,534 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=247 updating hbase:meta row=393f61e585ea6ccae9181b305adcbdf8, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:42,534 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088042534"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088042534"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088042534"}]},"ts":"1676088042534"} 2023-02-11 04:00:42,535 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=254, ppid=247, state=RUNNABLE; CloseRegionProcedure 393f61e585ea6ccae9181b305adcbdf8, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:42,682 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 73f6bdfd53354bbf7c9d40fd769c9e56 2023-02-11 04:00:42,682 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 73f6bdfd53354bbf7c9d40fd769c9e56, disabling compactions & flushes 2023-02-11 04:00:42,682 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. 2023-02-11 04:00:42,682 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. 2023-02-11 04:00:42,682 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. after waiting 0 ms 2023-02-11 04:00:42,682 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. 2023-02-11 04:00:42,682 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 73f6bdfd53354bbf7c9d40fd769c9e56 1/1 column families, dataSize=1.83 KB heapSize=4.19 KB 2023-02-11 04:00:42,683 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 660b743875c88bf2eaa58aab9da56413 2023-02-11 04:00:42,683 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 660b743875c88bf2eaa58aab9da56413, disabling compactions & flushes 2023-02-11 04:00:42,684 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413. 2023-02-11 04:00:42,684 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413. 2023-02-11 04:00:42,684 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413. after waiting 0 ms 2023-02-11 04:00:42,684 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413. 2023-02-11 04:00:42,684 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 660b743875c88bf2eaa58aab9da56413 1/1 column families, dataSize=2.22 KB heapSize=5.03 KB 2023-02-11 04:00:42,684 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a388dd856b0a069ab94fe6f6b9fb6351 2023-02-11 04:00:42,684 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing a388dd856b0a069ab94fe6f6b9fb6351, disabling compactions & flushes 2023-02-11 04:00:42,684 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351. 2023-02-11 04:00:42,684 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351. 2023-02-11 04:00:42,684 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351. after waiting 0 ms 2023-02-11 04:00:42,684 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351. 2023-02-11 04:00:42,684 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing a388dd856b0a069ab94fe6f6b9fb6351 1/1 column families, dataSize=2.42 KB heapSize=5.45 KB 2023-02-11 04:00:42,701 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230211d7f6567412ca47efb6bd3cbf605601b4 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/d41d8cd98f00b204e9800998ecf8427e20230211d7f6567412ca47efb6bd3cbf605601b4 2023-02-11 04:00:42,702 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b2023021131b6523e847f4a289e9b074bbe60ec35 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/c4ca4238a0b923820dcc509a6f75849b2023021131b6523e847f4a289e9b074bbe60ec35 2023-02-11 04:00:42,708 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56/.tmp/cf/2bb70252cfa54ef988c157f9aa816002 2023-02-11 04:00:42,709 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c2023021159ff96bbc053428e975bf9c29ce19ced to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/c81e728d9d4c2f636f067f89cc14862c2023021159ff96bbc053428e975bf9c29ce19ced 2023-02-11 04:00:42,711 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351/.tmp/cf/344f39904f8845f7bb49a0168e680fc3 2023-02-11 04:00:42,712 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56/.tmp/cf/2bb70252cfa54ef988c157f9aa816002 as hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56/cf/2bb70252cfa54ef988c157f9aa816002 2023-02-11 04:00:42,715 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351/.tmp/cf/344f39904f8845f7bb49a0168e680fc3 as hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351/cf/344f39904f8845f7bb49a0168e680fc3 2023-02-11 04:00:42,716 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56/cf/2bb70252cfa54ef988c157f9aa816002, entries=28, sequenceid=8, filesize=9.7 K 2023-02-11 04:00:42,717 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.83 KB/1874, heapSize ~4.17 KB/4272, currentSize=0 B/0 for 73f6bdfd53354bbf7c9d40fd769c9e56 in 35ms, sequenceid=8, compaction requested=false 2023-02-11 04:00:42,717 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testCorruptedSnapshot-1676088039685' 2023-02-11 04:00:42,719 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351/cf/344f39904f8845f7bb49a0168e680fc3, entries=37, sequenceid=8, filesize=11.3 K 2023-02-11 04:00:42,719 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.42 KB/2477, heapSize ~5.44 KB/5568, currentSize=0 B/0 for a388dd856b0a069ab94fe6f6b9fb6351 in 35ms, sequenceid=8, compaction requested=false 2023-02-11 04:00:42,721 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413/.tmp/cf/60aa41415b5e455e9c66d4ddf3b44e2c 2023-02-11 04:00:42,722 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 04:00:42,722 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. 2023-02-11 04:00:42,722 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 73f6bdfd53354bbf7c9d40fd769c9e56: Waiting for close lock at 1676088042682Running coprocessor pre-close hooks at 1676088042682Disabling compacts and flushes for region at 1676088042682Disabling writes for close at 1676088042682Obtaining lock to block concurrent updates at 1676088042682Preparing flush snapshotting stores in 73f6bdfd53354bbf7c9d40fd769c9e56 at 1676088042682Finished memstore snapshotting testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56., syncing WAL and waiting on mvcc, flushsize=dataSize=1874, getHeapSize=4272, getOffHeapSize=0, getCellsCount=28 at 1676088042682Flushing stores of testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. at 1676088042683 (+1 ms)Flushing 73f6bdfd53354bbf7c9d40fd769c9e56/cf: creating writer at 1676088042684 (+1 ms)Flushing mob file 73f6bdfd53354bbf7c9d40fd769c9e56/cf: appending metadata at 1676088042689 (+5 ms)Flushing mob file 73f6bdfd53354bbf7c9d40fd769c9e56/cf: closing flushed file at 1676088042689Flushing 73f6bdfd53354bbf7c9d40fd769c9e56/cf: appending metadata at 1676088042702 (+13 ms)Flushing 73f6bdfd53354bbf7c9d40fd769c9e56/cf: closing flushed file at 1676088042702Flushing 73f6bdfd53354bbf7c9d40fd769c9e56/cf: reopening flushed file at 1676088042713 (+11 ms)Finished flush of dataSize ~1.83 KB/1874, heapSize ~4.17 KB/4272, currentSize=0 B/0 for 73f6bdfd53354bbf7c9d40fd769c9e56 in 35ms, sequenceid=8, compaction requested=false at 1676088042717 (+4 ms)Writing region close event to WAL at 1676088042718 (+1 ms)Running coprocessor post-close hooks at 1676088042722 (+4 ms)Closed at 1676088042722 2023-02-11 04:00:42,724 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 73f6bdfd53354bbf7c9d40fd769c9e56 2023-02-11 04:00:42,724 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ee86f4ea9182f5cda2013084d379724f 2023-02-11 04:00:42,724 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing ee86f4ea9182f5cda2013084d379724f, disabling compactions & flushes 2023-02-11 04:00:42,724 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f. 2023-02-11 04:00:42,724 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f. 2023-02-11 04:00:42,724 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f. after waiting 0 ms 2023-02-11 04:00:42,724 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f. 2023-02-11 04:00:42,724 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 04:00:42,724 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413/.tmp/cf/60aa41415b5e455e9c66d4ddf3b44e2c as hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413/cf/60aa41415b5e455e9c66d4ddf3b44e2c 2023-02-11 04:00:42,724 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing ee86f4ea9182f5cda2013084d379724f 1/1 column families, dataSize=1.96 KB heapSize=4.47 KB 2023-02-11 04:00:42,724 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=243 updating hbase:meta row=73f6bdfd53354bbf7c9d40fd769c9e56, regionState=CLOSED 2023-02-11 04:00:42,725 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676088042724"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088042724"}]},"ts":"1676088042724"} 2023-02-11 04:00:42,725 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351. 2023-02-11 04:00:42,725 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for a388dd856b0a069ab94fe6f6b9fb6351: Waiting for close lock at 1676088042684Running coprocessor pre-close hooks at 1676088042684Disabling compacts and flushes for region at 1676088042684Disabling writes for close at 1676088042684Obtaining lock to block concurrent updates at 1676088042685 (+1 ms)Preparing flush snapshotting stores in a388dd856b0a069ab94fe6f6b9fb6351 at 1676088042685Finished memstore snapshotting testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351., syncing WAL and waiting on mvcc, flushsize=dataSize=2477, getHeapSize=5568, getOffHeapSize=0, getCellsCount=37 at 1676088042685Flushing stores of testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351. at 1676088042686 (+1 ms)Flushing a388dd856b0a069ab94fe6f6b9fb6351/cf: creating writer at 1676088042687 (+1 ms)Flushing mob file a388dd856b0a069ab94fe6f6b9fb6351/cf: appending metadata at 1676088042690 (+3 ms)Flushing mob file a388dd856b0a069ab94fe6f6b9fb6351/cf: closing flushed file at 1676088042690Flushing a388dd856b0a069ab94fe6f6b9fb6351/cf: appending metadata at 1676088042703 (+13 ms)Flushing a388dd856b0a069ab94fe6f6b9fb6351/cf: closing flushed file at 1676088042703Flushing a388dd856b0a069ab94fe6f6b9fb6351/cf: reopening flushed file at 1676088042715 (+12 ms)Finished flush of dataSize ~2.42 KB/2477, heapSize ~5.44 KB/5568, currentSize=0 B/0 for a388dd856b0a069ab94fe6f6b9fb6351 in 35ms, sequenceid=8, compaction requested=false at 1676088042719 (+4 ms)Writing region close event to WAL at 1676088042721 (+2 ms)Running coprocessor post-close hooks at 1676088042725 (+4 ms)Closed at 1676088042725 2023-02-11 04:00:42,727 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a388dd856b0a069ab94fe6f6b9fb6351 2023-02-11 04:00:42,727 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 998536da7be614a24a93f9e7ec31dbca 2023-02-11 04:00:42,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 998536da7be614a24a93f9e7ec31dbca, disabling compactions & flushes 2023-02-11 04:00:42,727 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca. 2023-02-11 04:00:42,727 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=244 updating hbase:meta row=a388dd856b0a069ab94fe6f6b9fb6351, regionState=CLOSED 2023-02-11 04:00:42,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca. 2023-02-11 04:00:42,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca. after waiting 0 ms 2023-02-11 04:00:42,728 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088042727"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088042727"}]},"ts":"1676088042727"} 2023-02-11 04:00:42,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca. 2023-02-11 04:00:42,728 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 998536da7be614a24a93f9e7ec31dbca 1/1 column families, dataSize=22.63 KB heapSize=48.91 KB 2023-02-11 04:00:42,730 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=253, resume processing ppid=243 2023-02-11 04:00:42,730 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=253, ppid=243, state=SUCCESS; CloseRegionProcedure 73f6bdfd53354bbf7c9d40fd769c9e56, server=jenkins-hbase4.apache.org,40829,1676088013739 in 194 msec 2023-02-11 04:00:42,731 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413/cf/60aa41415b5e455e9c66d4ddf3b44e2c, entries=34, sequenceid=8, filesize=10.8 K 2023-02-11 04:00:42,732 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.22 KB/2276, heapSize ~5.02 KB/5136, currentSize=0 B/0 for 660b743875c88bf2eaa58aab9da56413 in 48ms, sequenceid=8, compaction requested=false 2023-02-11 04:00:42,732 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=243, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=73f6bdfd53354bbf7c9d40fd769c9e56, UNASSIGN in 203 msec 2023-02-11 04:00:42,733 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=252, resume processing ppid=244 2023-02-11 04:00:42,734 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=252, ppid=244, state=SUCCESS; CloseRegionProcedure a388dd856b0a069ab94fe6f6b9fb6351, server=jenkins-hbase4.apache.org,42385,1676088013771 in 198 msec 2023-02-11 04:00:42,735 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=244, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=a388dd856b0a069ab94fe6f6b9fb6351, UNASSIGN in 206 msec 2023-02-11 04:00:42,738 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 04:00:42,739 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413. 2023-02-11 04:00:42,739 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 660b743875c88bf2eaa58aab9da56413: Waiting for close lock at 1676088042683Running coprocessor pre-close hooks at 1676088042683Disabling compacts and flushes for region at 1676088042683Disabling writes for close at 1676088042684 (+1 ms)Obtaining lock to block concurrent updates at 1676088042684Preparing flush snapshotting stores in 660b743875c88bf2eaa58aab9da56413 at 1676088042684Finished memstore snapshotting testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413., syncing WAL and waiting on mvcc, flushsize=dataSize=2276, getHeapSize=5136, getOffHeapSize=0, getCellsCount=34 at 1676088042684Flushing stores of testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413. at 1676088042685 (+1 ms)Flushing 660b743875c88bf2eaa58aab9da56413/cf: creating writer at 1676088042685Flushing mob file 660b743875c88bf2eaa58aab9da56413/cf: appending metadata at 1676088042690 (+5 ms)Flushing mob file 660b743875c88bf2eaa58aab9da56413/cf: closing flushed file at 1676088042690Flushing 660b743875c88bf2eaa58aab9da56413/cf: appending metadata at 1676088042710 (+20 ms)Flushing 660b743875c88bf2eaa58aab9da56413/cf: closing flushed file at 1676088042710Flushing 660b743875c88bf2eaa58aab9da56413/cf: reopening flushed file at 1676088042725 (+15 ms)Finished flush of dataSize ~2.22 KB/2276, heapSize ~5.02 KB/5136, currentSize=0 B/0 for 660b743875c88bf2eaa58aab9da56413 in 48ms, sequenceid=8, compaction requested=false at 1676088042732 (+7 ms)Writing region close event to WAL at 1676088042733 (+1 ms)Running coprocessor post-close hooks at 1676088042739 (+6 ms)Closed at 1676088042739 2023-02-11 04:00:42,740 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 660b743875c88bf2eaa58aab9da56413 2023-02-11 04:00:42,740 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 393f61e585ea6ccae9181b305adcbdf8 2023-02-11 04:00:42,740 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 393f61e585ea6ccae9181b305adcbdf8, disabling compactions & flushes 2023-02-11 04:00:42,740 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=245 updating hbase:meta row=660b743875c88bf2eaa58aab9da56413, regionState=CLOSED 2023-02-11 04:00:42,740 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8. 2023-02-11 04:00:42,740 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8. 2023-02-11 04:00:42,740 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088042740"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088042740"}]},"ts":"1676088042740"} 2023-02-11 04:00:42,740 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8. after waiting 0 ms 2023-02-11 04:00:42,741 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8. 2023-02-11 04:00:42,741 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 393f61e585ea6ccae9181b305adcbdf8 1/1 column families, dataSize=1.63 KB heapSize=3.77 KB 2023-02-11 04:00:42,745 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=250, resume processing ppid=245 2023-02-11 04:00:42,745 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=250, ppid=245, state=SUCCESS; CloseRegionProcedure 660b743875c88bf2eaa58aab9da56413, server=jenkins-hbase4.apache.org,46851,1676088013694 in 214 msec 2023-02-11 04:00:42,746 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=245, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=660b743875c88bf2eaa58aab9da56413, UNASSIGN in 218 msec 2023-02-11 04:00:42,760 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d520230211a94c568527944d96aba6533e72345c0f to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/e4da3b7fbbce2345d7772b0674a318d520230211a94c568527944d96aba6533e72345c0f 2023-02-11 04:00:42,760 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf320230211da45a7df0bf840a48ed16c26a4f5c52c to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211da45a7df0bf840a48ed16c26a4f5c52c 2023-02-11 04:00:42,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-11 04:00:42,775 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c20230211428491bd10e34f1cbf8b6789df0ee1c1 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/a87ff679a2f3e71d9181a67b7542122c20230211428491bd10e34f1cbf8b6789df0ee1c1 2023-02-11 04:00:42,784 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=22.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca/.tmp/cf/cc67fb5d7fb94be88db8bea101177d4b 2023-02-11 04:00:42,788 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca/.tmp/cf/cc67fb5d7fb94be88db8bea101177d4b as hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca/cf/cc67fb5d7fb94be88db8bea101177d4b 2023-02-11 04:00:42,790 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f/.tmp/cf/7b45a4d0a7ec4949ae74442957ffa428 2023-02-11 04:00:42,790 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca/cf/cc67fb5d7fb94be88db8bea101177d4b, entries=346, sequenceid=8, filesize=64.5 K 2023-02-11 04:00:42,791 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~22.63 KB/23178, heapSize ~48.89 KB/50064, currentSize=0 B/0 for 998536da7be614a24a93f9e7ec31dbca in 63ms, sequenceid=8, compaction requested=false 2023-02-11 04:00:42,793 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f/.tmp/cf/7b45a4d0a7ec4949ae74442957ffa428 as hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f/cf/7b45a4d0a7ec4949ae74442957ffa428 2023-02-11 04:00:42,794 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8/.tmp/cf/6da5642c1d0f4be18c5ac7efa0898774 2023-02-11 04:00:42,794 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 04:00:42,795 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca. 2023-02-11 04:00:42,795 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 998536da7be614a24a93f9e7ec31dbca: Waiting for close lock at 1676088042727Running coprocessor pre-close hooks at 1676088042727Disabling compacts and flushes for region at 1676088042727Disabling writes for close at 1676088042728 (+1 ms)Obtaining lock to block concurrent updates at 1676088042728Preparing flush snapshotting stores in 998536da7be614a24a93f9e7ec31dbca at 1676088042728Finished memstore snapshotting testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca., syncing WAL and waiting on mvcc, flushsize=dataSize=23178, getHeapSize=50064, getOffHeapSize=0, getCellsCount=346 at 1676088042728Flushing stores of testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca. at 1676088042729 (+1 ms)Flushing 998536da7be614a24a93f9e7ec31dbca/cf: creating writer at 1676088042730 (+1 ms)Flushing mob file 998536da7be614a24a93f9e7ec31dbca/cf: appending metadata at 1676088042737 (+7 ms)Flushing mob file 998536da7be614a24a93f9e7ec31dbca/cf: closing flushed file at 1676088042737Flushing 998536da7be614a24a93f9e7ec31dbca/cf: appending metadata at 1676088042761 (+24 ms)Flushing 998536da7be614a24a93f9e7ec31dbca/cf: closing flushed file at 1676088042761Flushing 998536da7be614a24a93f9e7ec31dbca/cf: reopening flushed file at 1676088042788 (+27 ms)Finished flush of dataSize ~22.63 KB/23178, heapSize ~48.89 KB/50064, currentSize=0 B/0 for 998536da7be614a24a93f9e7ec31dbca in 63ms, sequenceid=8, compaction requested=false at 1676088042791 (+3 ms)Writing region close event to WAL at 1676088042792 (+1 ms)Running coprocessor post-close hooks at 1676088042795 (+3 ms)Closed at 1676088042795 2023-02-11 04:00:42,796 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 998536da7be614a24a93f9e7ec31dbca 2023-02-11 04:00:42,796 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=248 updating hbase:meta row=998536da7be614a24a93f9e7ec31dbca, regionState=CLOSED 2023-02-11 04:00:42,796 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676088042796"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088042796"}]},"ts":"1676088042796"} 2023-02-11 04:00:42,797 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8/.tmp/cf/6da5642c1d0f4be18c5ac7efa0898774 as hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8/cf/6da5642c1d0f4be18c5ac7efa0898774 2023-02-11 04:00:42,799 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=251, resume processing ppid=248 2023-02-11 04:00:42,799 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=251, ppid=248, state=SUCCESS; CloseRegionProcedure 998536da7be614a24a93f9e7ec31dbca, server=jenkins-hbase4.apache.org,42385,1676088013771 in 267 msec 2023-02-11 04:00:42,800 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f/cf/7b45a4d0a7ec4949ae74442957ffa428, entries=30, sequenceid=8, filesize=10.1 K 2023-02-11 04:00:42,800 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8/cf/6da5642c1d0f4be18c5ac7efa0898774, entries=25, sequenceid=8, filesize=9.2 K 2023-02-11 04:00:42,801 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=248, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=998536da7be614a24a93f9e7ec31dbca, UNASSIGN in 272 msec 2023-02-11 04:00:42,801 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.63 KB/1673, heapSize ~3.75 KB/3840, currentSize=0 B/0 for 393f61e585ea6ccae9181b305adcbdf8 in 60ms, sequenceid=8, compaction requested=false 2023-02-11 04:00:42,801 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.96 KB/2008, heapSize ~4.45 KB/4560, currentSize=0 B/0 for ee86f4ea9182f5cda2013084d379724f in 77ms, sequenceid=8, compaction requested=false 2023-02-11 04:00:42,804 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 04:00:42,804 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8. 2023-02-11 04:00:42,804 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 393f61e585ea6ccae9181b305adcbdf8: Waiting for close lock at 1676088042740Running coprocessor pre-close hooks at 1676088042740Disabling compacts and flushes for region at 1676088042740Disabling writes for close at 1676088042741 (+1 ms)Obtaining lock to block concurrent updates at 1676088042741Preparing flush snapshotting stores in 393f61e585ea6ccae9181b305adcbdf8 at 1676088042741Finished memstore snapshotting testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8., syncing WAL and waiting on mvcc, flushsize=dataSize=1673, getHeapSize=3840, getOffHeapSize=0, getCellsCount=25 at 1676088042741Flushing stores of testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8. at 1676088042741Flushing 393f61e585ea6ccae9181b305adcbdf8/cf: creating writer at 1676088042744 (+3 ms)Flushing mob file 393f61e585ea6ccae9181b305adcbdf8/cf: appending metadata at 1676088042754 (+10 ms)Flushing mob file 393f61e585ea6ccae9181b305adcbdf8/cf: closing flushed file at 1676088042754Flushing 393f61e585ea6ccae9181b305adcbdf8/cf: appending metadata at 1676088042775 (+21 ms)Flushing 393f61e585ea6ccae9181b305adcbdf8/cf: closing flushed file at 1676088042776 (+1 ms)Flushing 393f61e585ea6ccae9181b305adcbdf8/cf: reopening flushed file at 1676088042798 (+22 ms)Finished flush of dataSize ~1.63 KB/1673, heapSize ~3.75 KB/3840, currentSize=0 B/0 for 393f61e585ea6ccae9181b305adcbdf8 in 60ms, sequenceid=8, compaction requested=false at 1676088042801 (+3 ms)Writing region close event to WAL at 1676088042801Running coprocessor post-close hooks at 1676088042804 (+3 ms)Closed at 1676088042804 2023-02-11 04:00:42,805 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 393f61e585ea6ccae9181b305adcbdf8 2023-02-11 04:00:42,805 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=247 updating hbase:meta row=393f61e585ea6ccae9181b305adcbdf8, regionState=CLOSED 2023-02-11 04:00:42,805 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088042805"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088042805"}]},"ts":"1676088042805"} 2023-02-11 04:00:42,808 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 04:00:42,808 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=254, resume processing ppid=247 2023-02-11 04:00:42,808 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f. 2023-02-11 04:00:42,808 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=254, ppid=247, state=SUCCESS; CloseRegionProcedure 393f61e585ea6ccae9181b305adcbdf8, server=jenkins-hbase4.apache.org,46851,1676088013694 in 272 msec 2023-02-11 04:00:42,808 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for ee86f4ea9182f5cda2013084d379724f: Waiting for close lock at 1676088042724Running coprocessor pre-close hooks at 1676088042724Disabling compacts and flushes for region at 1676088042724Disabling writes for close at 1676088042724Obtaining lock to block concurrent updates at 1676088042724Preparing flush snapshotting stores in ee86f4ea9182f5cda2013084d379724f at 1676088042724Finished memstore snapshotting testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f., syncing WAL and waiting on mvcc, flushsize=dataSize=2008, getHeapSize=4560, getOffHeapSize=0, getCellsCount=30 at 1676088042725 (+1 ms)Flushing stores of testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f. at 1676088042725Flushing ee86f4ea9182f5cda2013084d379724f/cf: creating writer at 1676088042727 (+2 ms)Flushing mob file ee86f4ea9182f5cda2013084d379724f/cf: appending metadata at 1676088042732 (+5 ms)Flushing mob file ee86f4ea9182f5cda2013084d379724f/cf: closing flushed file at 1676088042732Flushing ee86f4ea9182f5cda2013084d379724f/cf: appending metadata at 1676088042761 (+29 ms)Flushing ee86f4ea9182f5cda2013084d379724f/cf: closing flushed file at 1676088042761Flushing ee86f4ea9182f5cda2013084d379724f/cf: reopening flushed file at 1676088042794 (+33 ms)Finished flush of dataSize ~1.96 KB/2008, heapSize ~4.45 KB/4560, currentSize=0 B/0 for ee86f4ea9182f5cda2013084d379724f in 77ms, sequenceid=8, compaction requested=false at 1676088042801 (+7 ms)Writing region close event to WAL at 1676088042802 (+1 ms)Running coprocessor post-close hooks at 1676088042808 (+6 ms)Closed at 1676088042808 2023-02-11 04:00:42,809 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=247, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=393f61e585ea6ccae9181b305adcbdf8, UNASSIGN in 281 msec 2023-02-11 04:00:42,810 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ee86f4ea9182f5cda2013084d379724f 2023-02-11 04:00:42,810 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=246 updating hbase:meta row=ee86f4ea9182f5cda2013084d379724f, regionState=CLOSED 2023-02-11 04:00:42,810 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088042810"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088042810"}]},"ts":"1676088042810"} 2023-02-11 04:00:42,812 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=249, resume processing ppid=246 2023-02-11 04:00:42,812 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=249, ppid=246, state=SUCCESS; CloseRegionProcedure ee86f4ea9182f5cda2013084d379724f, server=jenkins-hbase4.apache.org,40829,1676088013739 in 281 msec 2023-02-11 04:00:42,814 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=246, resume processing ppid=242 2023-02-11 04:00:42,814 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=246, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=ee86f4ea9182f5cda2013084d379724f, UNASSIGN in 285 msec 2023-02-11 04:00:42,814 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676088039685","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088042814"}]},"ts":"1676088042814"} 2023-02-11 04:00:42,815 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676088039685, state=DISABLED in hbase:meta 2023-02-11 04:00:42,817 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676088039685 to state=DISABLED 2023-02-11 04:00:42,819 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=242, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676088039685 in 298 msec 2023-02-11 04:00:43,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-11 04:00:43,275 INFO [Listener at localhost/33753] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676088039685, procId: 242 completed 2023-02-11 04:00:43,276 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=snaptb0-1676088039685 table=testCorruptedSnapshot-1676088039685 type=FLUSH ttl=0 } 2023-02-11 04:00:43,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676088043276 (current time:1676088043276). 2023-02-11 04:00:43,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-11 04:00:43,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] zookeeper.ReadOnlyZKClient(139): Connect 0x7f97e1bd to 127.0.0.1:63425 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 04:00:43,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6c0d331, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 04:00:43,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 04:00:43,286 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39112, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 04:00:43,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7f97e1bd to 127.0.0.1:63425 2023-02-11 04:00:43,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:00:43,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-11 04:00:43,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb0-1676088039685 table=testCorruptedSnapshot-1676088039685 type=FLUSH ttl=0 } 2023-02-11 04:00:43,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] procedure2.ProcedureExecutor(1029): Stored pid=255, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676088039685, type=EXCLUSIVE 2023-02-11 04:00:43,290 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=255, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676088039685, type=EXCLUSIVE 2023-02-11 04:00:43,291 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=255, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676088039685, type=EXCLUSIVE; timeout=600000, timestamp=1676088643290 2023-02-11 04:00:43,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb0-1676088039685 table=testCorruptedSnapshot-1676088039685 type=FLUSH ttl=0 } 2023-02-11 04:00:43,291 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb0-1676088039685 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676088039685 2023-02-11 04:00:43,291 DEBUG [Listener at localhost/33753] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb0-1676088039685 table=testCorruptedSnapshot-1676088039685 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-11 04:00:43,291 DEBUG [Listener at localhost/33753] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-11 04:00:43,299 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676088039685 table=testCorruptedSnapshot-1676088039685 type=DISABLED ttl=0 } 2023-02-11 04:00:43,300 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:43,300 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:43,300 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:43,301 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:43,301 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:43,301 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:43,301 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:43,301 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:43,301 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:43,301 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:43,301 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:43,301 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:43,302 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 04:00:43,302 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 04:00:43,303 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56/cf/2bb70252cfa54ef988c157f9aa816002] hfiles 2023-02-11 04:00:43,303 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56/cf/2bb70252cfa54ef988c157f9aa816002 2023-02-11 04:00:43,303 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8/cf/6da5642c1d0f4be18c5ac7efa0898774] hfiles 2023-02-11 04:00:43,303 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8/cf/6da5642c1d0f4be18c5ac7efa0898774 2023-02-11 04:00:43,303 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f/cf/7b45a4d0a7ec4949ae74442957ffa428] hfiles 2023-02-11 04:00:43,303 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f/cf/7b45a4d0a7ec4949ae74442957ffa428 2023-02-11 04:00:43,304 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca/cf/cc67fb5d7fb94be88db8bea101177d4b] hfiles 2023-02-11 04:00:43,304 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca/cf/cc67fb5d7fb94be88db8bea101177d4b 2023-02-11 04:00:43,304 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351/cf/344f39904f8845f7bb49a0168e680fc3] hfiles 2023-02-11 04:00:43,304 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351/cf/344f39904f8845f7bb49a0168e680fc3 2023-02-11 04:00:43,304 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413/cf/60aa41415b5e455e9c66d4ddf3b44e2c] hfiles 2023-02-11 04:00:43,305 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413/cf/60aa41415b5e455e9c66d4ddf3b44e2c 2023-02-11 04:00:43,307 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/a87ff679a2f3e71d9181a67b7542122c20230211428491bd10e34f1cbf8b6789df0ee1c1, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/c4ca4238a0b923820dcc509a6f75849b2023021131b6523e847f4a289e9b074bbe60ec35, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/c81e728d9d4c2f636f067f89cc14862c2023021159ff96bbc053428e975bf9c29ce19ced, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/d41d8cd98f00b204e9800998ecf8427e20230211d7f6567412ca47efb6bd3cbf605601b4, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/e4da3b7fbbce2345d7772b0674a318d520230211a94c568527944d96aba6533e72345c0f, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211da45a7df0bf840a48ed16c26a4f5c52c] hfiles 2023-02-11 04:00:43,307 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/a87ff679a2f3e71d9181a67b7542122c20230211428491bd10e34f1cbf8b6789df0ee1c1 2023-02-11 04:00:43,307 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/c4ca4238a0b923820dcc509a6f75849b2023021131b6523e847f4a289e9b074bbe60ec35 2023-02-11 04:00:43,307 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/c81e728d9d4c2f636f067f89cc14862c2023021159ff96bbc053428e975bf9c29ce19ced 2023-02-11 04:00:43,307 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/d41d8cd98f00b204e9800998ecf8427e20230211d7f6567412ca47efb6bd3cbf605601b4 2023-02-11 04:00:43,307 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/e4da3b7fbbce2345d7772b0674a318d520230211a94c568527944d96aba6533e72345c0f 2023-02-11 04:00:43,307 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211da45a7df0bf840a48ed16c26a4f5c52c 2023-02-11 04:00:43,321 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb0-1676088039685 table=testCorruptedSnapshot-1676088039685 type=DISABLED ttl=0 } as finished. 2023-02-11 04:00:43,321 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb0-1676088039685 2023-02-11 04:00:43,322 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/.tmp/snaptb0-1676088039685 2023-02-11 04:00:43,338 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/.tmp/snaptb0-1676088039685 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/snaptb0-1676088039685 2023-02-11 04:00:43,356 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb0-1676088039685 of table testCorruptedSnapshot-1676088039685 completed 2023-02-11 04:00:43,356 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/.tmp/snaptb0-1676088039685 2023-02-11 04:00:43,356 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.hbase-snapshot/.tmp/snaptb0-1676088039685 2023-02-11 04:00:43,356 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb0-1676088039685 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676088039685 at 1676088043291Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676088039685 table=testCorruptedSnapshot-1676088039685 type=DISABLED ttl=0 } at 1676088043299 (+8 ms)Consolidate snapshot: snaptb0-1676088039685 at 1676088043321 (+22 ms)Loading Region manifests for snaptb0-1676088039685 at 1676088043321Writing data manifest for snaptb0-1676088039685 at 1676088043324 (+3 ms)Verifying snapshot: snaptb0-1676088039685 at 1676088043331 (+7 ms)Snapshot snaptb0-1676088039685 of table testCorruptedSnapshot-1676088039685 completed at 1676088043356 (+25 ms) 2023-02-11 04:00:43,357 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=255, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676088039685, type=EXCLUSIVE 2023-02-11 04:00:43,358 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=255, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676088039685, type=EXCLUSIVE in 68 msec 2023-02-11 04:00:43,541 DEBUG [Listener at localhost/33753] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 04:00:43,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676088039685 table=testCorruptedSnapshot-1676088039685 type=FLUSH ttl=0 } is done 2023-02-11 04:00:43,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb0-1676088039685 table=testCorruptedSnapshot-1676088039685 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-11 04:00:43,543 INFO [Listener at localhost/33753] client.HBaseAdmin$14(830): Started enable of testCorruptedSnapshot-1676088039685 2023-02-11 04:00:43,543 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testCorruptedSnapshot-1676088039685 2023-02-11 04:00:43,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] procedure2.ProcedureExecutor(1029): Stored pid=256, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testCorruptedSnapshot-1676088039685 2023-02-11 04:00:43,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-11 04:00:43,548 INFO [PEWorker-2] procedure.EnableTableProcedure(296): Attempting to enable the table testCorruptedSnapshot-1676088039685 2023-02-11 04:00:43,548 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676088039685","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088043548"}]},"ts":"1676088043548"} 2023-02-11 04:00:43,549 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676088039685, state=ENABLING in hbase:meta 2023-02-11 04:00:43,551 DEBUG [PEWorker-2] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-11 04:00:43,552 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=73f6bdfd53354bbf7c9d40fd769c9e56, ASSIGN}, {pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=a388dd856b0a069ab94fe6f6b9fb6351, ASSIGN}, {pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=660b743875c88bf2eaa58aab9da56413, ASSIGN}, {pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=ee86f4ea9182f5cda2013084d379724f, ASSIGN}, {pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=393f61e585ea6ccae9181b305adcbdf8, ASSIGN}, {pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=998536da7be614a24a93f9e7ec31dbca, ASSIGN}] 2023-02-11 04:00:43,552 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=ee86f4ea9182f5cda2013084d379724f, ASSIGN 2023-02-11 04:00:43,552 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=a388dd856b0a069ab94fe6f6b9fb6351, ASSIGN 2023-02-11 04:00:43,553 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=660b743875c88bf2eaa58aab9da56413, ASSIGN 2023-02-11 04:00:43,553 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=73f6bdfd53354bbf7c9d40fd769c9e56, ASSIGN 2023-02-11 04:00:43,553 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:43,553 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:43,553 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=ee86f4ea9182f5cda2013084d379724f, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,40829,1676088013739; forceNewPlan=false, retain=true 2023-02-11 04:00:43,553 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=a388dd856b0a069ab94fe6f6b9fb6351, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,42385,1676088013771; forceNewPlan=false, retain=true 2023-02-11 04:00:43,554 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:43,554 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=998536da7be614a24a93f9e7ec31dbca, ASSIGN 2023-02-11 04:00:43,554 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=660b743875c88bf2eaa58aab9da56413, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,46851,1676088013694; forceNewPlan=false, retain=true 2023-02-11 04:00:43,554 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:43,554 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=393f61e585ea6ccae9181b305adcbdf8, ASSIGN 2023-02-11 04:00:43,554 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=73f6bdfd53354bbf7c9d40fd769c9e56, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,40829,1676088013739; forceNewPlan=false, retain=true 2023-02-11 04:00:43,555 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:43,555 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:43,555 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=998536da7be614a24a93f9e7ec31dbca, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,42385,1676088013771; forceNewPlan=false, retain=true 2023-02-11 04:00:43,555 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=393f61e585ea6ccae9181b305adcbdf8, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,46851,1676088013694; forceNewPlan=false, retain=true 2023-02-11 04:00:43,704 INFO [jenkins-hbase4:41713] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-11 04:00:43,704 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=259 updating hbase:meta row=660b743875c88bf2eaa58aab9da56413, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:43,704 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=260 updating hbase:meta row=ee86f4ea9182f5cda2013084d379724f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:43,704 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=258 updating hbase:meta row=a388dd856b0a069ab94fe6f6b9fb6351, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:43,704 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088043704"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088043704"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088043704"}]},"ts":"1676088043704"} 2023-02-11 04:00:43,704 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=262 updating hbase:meta row=998536da7be614a24a93f9e7ec31dbca, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:43,704 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=261 updating hbase:meta row=393f61e585ea6ccae9181b305adcbdf8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:43,704 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676088043704"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088043704"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088043704"}]},"ts":"1676088043704"} 2023-02-11 04:00:43,704 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088043704"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088043704"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088043704"}]},"ts":"1676088043704"} 2023-02-11 04:00:43,704 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088043704"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088043704"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088043704"}]},"ts":"1676088043704"} 2023-02-11 04:00:43,704 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088043704"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088043704"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088043704"}]},"ts":"1676088043704"} 2023-02-11 04:00:43,705 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=263, ppid=259, state=RUNNABLE; OpenRegionProcedure 660b743875c88bf2eaa58aab9da56413, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:43,706 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=264, ppid=262, state=RUNNABLE; OpenRegionProcedure 998536da7be614a24a93f9e7ec31dbca, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:43,707 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=265, ppid=258, state=RUNNABLE; OpenRegionProcedure a388dd856b0a069ab94fe6f6b9fb6351, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:43,707 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=266, ppid=260, state=RUNNABLE; OpenRegionProcedure ee86f4ea9182f5cda2013084d379724f, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:43,708 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=267, ppid=261, state=RUNNABLE; OpenRegionProcedure 393f61e585ea6ccae9181b305adcbdf8, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:43,709 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=257 updating hbase:meta row=73f6bdfd53354bbf7c9d40fd769c9e56, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:43,709 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676088043709"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088043709"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088043709"}]},"ts":"1676088043709"} 2023-02-11 04:00:43,710 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=268, ppid=257, state=RUNNABLE; OpenRegionProcedure 73f6bdfd53354bbf7c9d40fd769c9e56, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:43,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-11 04:00:43,859 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8. 2023-02-11 04:00:43,859 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca. 2023-02-11 04:00:43,859 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 393f61e585ea6ccae9181b305adcbdf8, NAME => 'testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 04:00:43,860 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 998536da7be614a24a93f9e7ec31dbca, NAME => 'testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 04:00:43,860 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676088039685 393f61e585ea6ccae9181b305adcbdf8 2023-02-11 04:00:43,860 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676088039685 998536da7be614a24a93f9e7ec31dbca 2023-02-11 04:00:43,860 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:43,860 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:43,860 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 393f61e585ea6ccae9181b305adcbdf8 2023-02-11 04:00:43,860 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 998536da7be614a24a93f9e7ec31dbca 2023-02-11 04:00:43,860 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 998536da7be614a24a93f9e7ec31dbca 2023-02-11 04:00:43,860 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 393f61e585ea6ccae9181b305adcbdf8 2023-02-11 04:00:43,861 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f. 2023-02-11 04:00:43,861 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ee86f4ea9182f5cda2013084d379724f, NAME => 'testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 04:00:43,861 INFO [StoreOpener-393f61e585ea6ccae9181b305adcbdf8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 393f61e585ea6ccae9181b305adcbdf8 2023-02-11 04:00:43,861 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676088039685 ee86f4ea9182f5cda2013084d379724f 2023-02-11 04:00:43,861 INFO [StoreOpener-998536da7be614a24a93f9e7ec31dbca-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 998536da7be614a24a93f9e7ec31dbca 2023-02-11 04:00:43,861 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:43,861 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for ee86f4ea9182f5cda2013084d379724f 2023-02-11 04:00:43,861 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for ee86f4ea9182f5cda2013084d379724f 2023-02-11 04:00:43,862 DEBUG [StoreOpener-998536da7be614a24a93f9e7ec31dbca-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca/cf 2023-02-11 04:00:43,862 DEBUG [StoreOpener-998536da7be614a24a93f9e7ec31dbca-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca/cf 2023-02-11 04:00:43,862 DEBUG [StoreOpener-393f61e585ea6ccae9181b305adcbdf8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8/cf 2023-02-11 04:00:43,862 DEBUG [StoreOpener-393f61e585ea6ccae9181b305adcbdf8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8/cf 2023-02-11 04:00:43,862 INFO [StoreOpener-ee86f4ea9182f5cda2013084d379724f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ee86f4ea9182f5cda2013084d379724f 2023-02-11 04:00:43,863 INFO [StoreOpener-998536da7be614a24a93f9e7ec31dbca-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 998536da7be614a24a93f9e7ec31dbca columnFamilyName cf 2023-02-11 04:00:43,863 INFO [StoreOpener-393f61e585ea6ccae9181b305adcbdf8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 393f61e585ea6ccae9181b305adcbdf8 columnFamilyName cf 2023-02-11 04:00:43,863 DEBUG [StoreOpener-ee86f4ea9182f5cda2013084d379724f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f/cf 2023-02-11 04:00:43,863 DEBUG [StoreOpener-ee86f4ea9182f5cda2013084d379724f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f/cf 2023-02-11 04:00:43,864 INFO [StoreOpener-ee86f4ea9182f5cda2013084d379724f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ee86f4ea9182f5cda2013084d379724f columnFamilyName cf 2023-02-11 04:00:43,866 DEBUG [StoreOpener-393f61e585ea6ccae9181b305adcbdf8-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8/cf/6da5642c1d0f4be18c5ac7efa0898774 2023-02-11 04:00:43,866 DEBUG [StoreOpener-998536da7be614a24a93f9e7ec31dbca-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca/cf/cc67fb5d7fb94be88db8bea101177d4b 2023-02-11 04:00:43,866 INFO [StoreOpener-393f61e585ea6ccae9181b305adcbdf8-1] regionserver.HStore(310): Store=393f61e585ea6ccae9181b305adcbdf8/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:43,867 INFO [StoreOpener-998536da7be614a24a93f9e7ec31dbca-1] regionserver.HStore(310): Store=998536da7be614a24a93f9e7ec31dbca/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:43,867 DEBUG [StoreOpener-ee86f4ea9182f5cda2013084d379724f-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f/cf/7b45a4d0a7ec4949ae74442957ffa428 2023-02-11 04:00:43,867 INFO [StoreOpener-ee86f4ea9182f5cda2013084d379724f-1] regionserver.HStore(310): Store=ee86f4ea9182f5cda2013084d379724f/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:43,867 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8 2023-02-11 04:00:43,867 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca 2023-02-11 04:00:43,868 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f 2023-02-11 04:00:43,868 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8 2023-02-11 04:00:43,868 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca 2023-02-11 04:00:43,868 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f 2023-02-11 04:00:43,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 393f61e585ea6ccae9181b305adcbdf8 2023-02-11 04:00:43,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 998536da7be614a24a93f9e7ec31dbca 2023-02-11 04:00:43,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for ee86f4ea9182f5cda2013084d379724f 2023-02-11 04:00:43,871 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 393f61e585ea6ccae9181b305adcbdf8; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71326135, jitterRate=0.0628422349691391}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:43,871 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened ee86f4ea9182f5cda2013084d379724f; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74030335, jitterRate=0.10313795506954193}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:43,871 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 998536da7be614a24a93f9e7ec31dbca; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61475871, jitterRate=-0.08393813669681549}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:43,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for ee86f4ea9182f5cda2013084d379724f: Running coprocessor pre-open hook at 1676088043861Writing region info on filesystem at 1676088043861Initializing all the Stores at 1676088043862 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088043862Cleaning up temporary data from old regions at 1676088043869 (+7 ms)Cleaning up detritus from prior splits at 1676088043869Running coprocessor post-open hooks at 1676088043871 (+2 ms)Region opened successfully at 1676088043871 2023-02-11 04:00:43,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 998536da7be614a24a93f9e7ec31dbca: Running coprocessor pre-open hook at 1676088043860Writing region info on filesystem at 1676088043860Initializing all the Stores at 1676088043861 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088043861Cleaning up temporary data from old regions at 1676088043869 (+8 ms)Cleaning up detritus from prior splits at 1676088043869Running coprocessor post-open hooks at 1676088043871 (+2 ms)Region opened successfully at 1676088043871 2023-02-11 04:00:43,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 393f61e585ea6ccae9181b305adcbdf8: Running coprocessor pre-open hook at 1676088043860Writing region info on filesystem at 1676088043860Initializing all the Stores at 1676088043861 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088043861Cleaning up temporary data from old regions at 1676088043868 (+7 ms)Cleaning up detritus from prior splits at 1676088043869 (+1 ms)Running coprocessor post-open hooks at 1676088043871 (+2 ms)Region opened successfully at 1676088043871 2023-02-11 04:00:43,872 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca., pid=264, masterSystemTime=1676088043857 2023-02-11 04:00:43,872 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f., pid=266, masterSystemTime=1676088043859 2023-02-11 04:00:43,872 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8., pid=267, masterSystemTime=1676088043857 2023-02-11 04:00:43,873 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca. 2023-02-11 04:00:43,873 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca. 2023-02-11 04:00:43,873 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351. 2023-02-11 04:00:43,873 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a388dd856b0a069ab94fe6f6b9fb6351, NAME => 'testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 04:00:43,873 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=262 updating hbase:meta row=998536da7be614a24a93f9e7ec31dbca, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:43,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676088039685 a388dd856b0a069ab94fe6f6b9fb6351 2023-02-11 04:00:43,874 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676088043873"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088043873"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088043873"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088043873"}]},"ts":"1676088043873"} 2023-02-11 04:00:43,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f. 2023-02-11 04:00:43,874 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f. 2023-02-11 04:00:43,874 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. 2023-02-11 04:00:43,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:43,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 73f6bdfd53354bbf7c9d40fd769c9e56, NAME => 'testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 04:00:43,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for a388dd856b0a069ab94fe6f6b9fb6351 2023-02-11 04:00:43,874 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=260 updating hbase:meta row=ee86f4ea9182f5cda2013084d379724f, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:43,874 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088043874"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088043874"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088043874"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088043874"}]},"ts":"1676088043874"} 2023-02-11 04:00:43,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for a388dd856b0a069ab94fe6f6b9fb6351 2023-02-11 04:00:43,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676088039685 73f6bdfd53354bbf7c9d40fd769c9e56 2023-02-11 04:00:43,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:43,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8. 2023-02-11 04:00:43,875 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 73f6bdfd53354bbf7c9d40fd769c9e56 2023-02-11 04:00:43,875 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8. 2023-02-11 04:00:43,875 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=261 updating hbase:meta row=393f61e585ea6ccae9181b305adcbdf8, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:43,875 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 73f6bdfd53354bbf7c9d40fd769c9e56 2023-02-11 04:00:43,875 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413. 2023-02-11 04:00:43,875 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088043875"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088043875"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088043875"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088043875"}]},"ts":"1676088043875"} 2023-02-11 04:00:43,875 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 660b743875c88bf2eaa58aab9da56413, NAME => 'testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 04:00:43,875 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676088039685 660b743875c88bf2eaa58aab9da56413 2023-02-11 04:00:43,875 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 04:00:43,875 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 660b743875c88bf2eaa58aab9da56413 2023-02-11 04:00:43,875 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 660b743875c88bf2eaa58aab9da56413 2023-02-11 04:00:43,875 INFO [StoreOpener-a388dd856b0a069ab94fe6f6b9fb6351-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a388dd856b0a069ab94fe6f6b9fb6351 2023-02-11 04:00:43,876 INFO [StoreOpener-73f6bdfd53354bbf7c9d40fd769c9e56-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 73f6bdfd53354bbf7c9d40fd769c9e56 2023-02-11 04:00:43,876 INFO [StoreOpener-660b743875c88bf2eaa58aab9da56413-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 660b743875c88bf2eaa58aab9da56413 2023-02-11 04:00:43,876 DEBUG [StoreOpener-a388dd856b0a069ab94fe6f6b9fb6351-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351/cf 2023-02-11 04:00:43,877 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=264, resume processing ppid=262 2023-02-11 04:00:43,877 DEBUG [StoreOpener-a388dd856b0a069ab94fe6f6b9fb6351-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351/cf 2023-02-11 04:00:43,877 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=264, ppid=262, state=SUCCESS; OpenRegionProcedure 998536da7be614a24a93f9e7ec31dbca, server=jenkins-hbase4.apache.org,42385,1676088013771 in 169 msec 2023-02-11 04:00:43,877 INFO [StoreOpener-a388dd856b0a069ab94fe6f6b9fb6351-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a388dd856b0a069ab94fe6f6b9fb6351 columnFamilyName cf 2023-02-11 04:00:43,877 DEBUG [StoreOpener-660b743875c88bf2eaa58aab9da56413-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413/cf 2023-02-11 04:00:43,877 DEBUG [StoreOpener-660b743875c88bf2eaa58aab9da56413-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413/cf 2023-02-11 04:00:43,877 DEBUG [StoreOpener-73f6bdfd53354bbf7c9d40fd769c9e56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56/cf 2023-02-11 04:00:43,877 DEBUG [StoreOpener-73f6bdfd53354bbf7c9d40fd769c9e56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56/cf 2023-02-11 04:00:43,878 INFO [StoreOpener-660b743875c88bf2eaa58aab9da56413-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 660b743875c88bf2eaa58aab9da56413 columnFamilyName cf 2023-02-11 04:00:43,878 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=266, resume processing ppid=260 2023-02-11 04:00:43,878 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=266, ppid=260, state=SUCCESS; OpenRegionProcedure ee86f4ea9182f5cda2013084d379724f, server=jenkins-hbase4.apache.org,40829,1676088013739 in 169 msec 2023-02-11 04:00:43,878 INFO [StoreOpener-73f6bdfd53354bbf7c9d40fd769c9e56-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 73f6bdfd53354bbf7c9d40fd769c9e56 columnFamilyName cf 2023-02-11 04:00:43,878 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=262, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=998536da7be614a24a93f9e7ec31dbca, ASSIGN in 325 msec 2023-02-11 04:00:43,879 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=267, resume processing ppid=261 2023-02-11 04:00:43,879 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=267, ppid=261, state=SUCCESS; OpenRegionProcedure 393f61e585ea6ccae9181b305adcbdf8, server=jenkins-hbase4.apache.org,46851,1676088013694 in 169 msec 2023-02-11 04:00:43,879 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=260, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=ee86f4ea9182f5cda2013084d379724f, ASSIGN in 326 msec 2023-02-11 04:00:43,880 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=261, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=393f61e585ea6ccae9181b305adcbdf8, ASSIGN in 327 msec 2023-02-11 04:00:43,881 DEBUG [StoreOpener-a388dd856b0a069ab94fe6f6b9fb6351-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351/cf/344f39904f8845f7bb49a0168e680fc3 2023-02-11 04:00:43,882 INFO [StoreOpener-a388dd856b0a069ab94fe6f6b9fb6351-1] regionserver.HStore(310): Store=a388dd856b0a069ab94fe6f6b9fb6351/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:43,882 DEBUG [StoreOpener-660b743875c88bf2eaa58aab9da56413-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413/cf/60aa41415b5e455e9c66d4ddf3b44e2c 2023-02-11 04:00:43,882 INFO [StoreOpener-660b743875c88bf2eaa58aab9da56413-1] regionserver.HStore(310): Store=660b743875c88bf2eaa58aab9da56413/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:43,882 DEBUG [StoreOpener-73f6bdfd53354bbf7c9d40fd769c9e56-1] regionserver.HStore(539): loaded hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56/cf/2bb70252cfa54ef988c157f9aa816002 2023-02-11 04:00:43,882 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351 2023-02-11 04:00:43,882 INFO [StoreOpener-73f6bdfd53354bbf7c9d40fd769c9e56-1] regionserver.HStore(310): Store=73f6bdfd53354bbf7c9d40fd769c9e56/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 04:00:43,883 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413 2023-02-11 04:00:43,883 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56 2023-02-11 04:00:43,883 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351 2023-02-11 04:00:43,883 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413 2023-02-11 04:00:43,884 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56 2023-02-11 04:00:43,885 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for a388dd856b0a069ab94fe6f6b9fb6351 2023-02-11 04:00:43,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 660b743875c88bf2eaa58aab9da56413 2023-02-11 04:00:43,888 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened a388dd856b0a069ab94fe6f6b9fb6351; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75490060, jitterRate=0.1248895525932312}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:43,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for a388dd856b0a069ab94fe6f6b9fb6351: Running coprocessor pre-open hook at 1676088043874Writing region info on filesystem at 1676088043874Initializing all the Stores at 1676088043875 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088043875Cleaning up temporary data from old regions at 1676088043883 (+8 ms)Cleaning up detritus from prior splits at 1676088043884 (+1 ms)Running coprocessor post-open hooks at 1676088043888 (+4 ms)Region opened successfully at 1676088043888 2023-02-11 04:00:43,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 73f6bdfd53354bbf7c9d40fd769c9e56 2023-02-11 04:00:43,889 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 660b743875c88bf2eaa58aab9da56413; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74070035, jitterRate=0.10372953116893768}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:43,889 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 660b743875c88bf2eaa58aab9da56413: Running coprocessor pre-open hook at 1676088043875Writing region info on filesystem at 1676088043875Initializing all the Stores at 1676088043876 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088043876Cleaning up temporary data from old regions at 1676088043884 (+8 ms)Cleaning up detritus from prior splits at 1676088043884Running coprocessor post-open hooks at 1676088043889 (+5 ms)Region opened successfully at 1676088043889 2023-02-11 04:00:43,889 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 73f6bdfd53354bbf7c9d40fd769c9e56; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69539107, jitterRate=0.03621344268321991}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 04:00:43,889 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351., pid=265, masterSystemTime=1676088043857 2023-02-11 04:00:43,889 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413., pid=263, masterSystemTime=1676088043857 2023-02-11 04:00:43,889 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 73f6bdfd53354bbf7c9d40fd769c9e56: Running coprocessor pre-open hook at 1676088043875Writing region info on filesystem at 1676088043875Initializing all the Stores at 1676088043875Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676088043875Cleaning up temporary data from old regions at 1676088043884 (+9 ms)Cleaning up detritus from prior splits at 1676088043884Running coprocessor post-open hooks at 1676088043889 (+5 ms)Region opened successfully at 1676088043889 2023-02-11 04:00:43,890 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56., pid=268, masterSystemTime=1676088043859 2023-02-11 04:00:43,890 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351. 2023-02-11 04:00:43,890 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351. 2023-02-11 04:00:43,891 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=258 updating hbase:meta row=a388dd856b0a069ab94fe6f6b9fb6351, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:43,891 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413. 2023-02-11 04:00:43,891 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088043891"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088043891"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088043891"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088043891"}]},"ts":"1676088043891"} 2023-02-11 04:00:43,891 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413. 2023-02-11 04:00:43,891 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=259 updating hbase:meta row=660b743875c88bf2eaa58aab9da56413, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:43,891 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088043891"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088043891"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088043891"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088043891"}]},"ts":"1676088043891"} 2023-02-11 04:00:43,892 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. 2023-02-11 04:00:43,892 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. 2023-02-11 04:00:43,892 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=257 updating hbase:meta row=73f6bdfd53354bbf7c9d40fd769c9e56, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:43,892 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676088043892"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676088043892"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676088043892"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676088043892"}]},"ts":"1676088043892"} 2023-02-11 04:00:43,894 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=265, resume processing ppid=258 2023-02-11 04:00:43,894 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=265, ppid=258, state=SUCCESS; OpenRegionProcedure a388dd856b0a069ab94fe6f6b9fb6351, server=jenkins-hbase4.apache.org,42385,1676088013771 in 185 msec 2023-02-11 04:00:43,895 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=263, resume processing ppid=259 2023-02-11 04:00:43,895 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=263, ppid=259, state=SUCCESS; OpenRegionProcedure 660b743875c88bf2eaa58aab9da56413, server=jenkins-hbase4.apache.org,46851,1676088013694 in 188 msec 2023-02-11 04:00:43,896 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=268, resume processing ppid=257 2023-02-11 04:00:43,896 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=258, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=a388dd856b0a069ab94fe6f6b9fb6351, ASSIGN in 343 msec 2023-02-11 04:00:43,896 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=268, ppid=257, state=SUCCESS; OpenRegionProcedure 73f6bdfd53354bbf7c9d40fd769c9e56, server=jenkins-hbase4.apache.org,40829,1676088013739 in 183 msec 2023-02-11 04:00:43,896 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=259, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=660b743875c88bf2eaa58aab9da56413, ASSIGN in 343 msec 2023-02-11 04:00:43,897 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=257, resume processing ppid=256 2023-02-11 04:00:43,897 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=257, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=73f6bdfd53354bbf7c9d40fd769c9e56, ASSIGN in 345 msec 2023-02-11 04:00:43,897 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676088039685","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088043897"}]},"ts":"1676088043897"} 2023-02-11 04:00:43,898 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676088039685, state=ENABLED in hbase:meta 2023-02-11 04:00:43,900 INFO [PEWorker-4] procedure.EnableTableProcedure(310): Table 'testCorruptedSnapshot-1676088039685' was successfully enabled. 2023-02-11 04:00:43,901 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=256, state=SUCCESS; EnableTableProcedure table=testCorruptedSnapshot-1676088039685 in 357 msec 2023-02-11 04:00:44,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-11 04:00:44,298 INFO [Listener at localhost/33753] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testCorruptedSnapshot-1676088039685, procId: 256 completed 2023-02-11 04:00:44,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 04:00:44,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 04:00:44,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40829] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 04:00:44,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46851] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 04:00:44,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 04:00:44,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42385] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 04:00:44,311 DEBUG [Listener at localhost/33753] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676088039685 2023-02-11 04:00:44,311 DEBUG [Listener at localhost/33753] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. 2023-02-11 04:00:44,312 INFO [Listener at localhost/33753] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-11 04:00:44,399 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x69705b98 to 127.0.0.1:63425 2023-02-11 04:00:44,399 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:00:44,440 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 04:00:44,524 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 04:00:44,599 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 04:00:44,666 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 04:00:44,725 DEBUG [Listener at localhost/33753] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 04:00:45,544 ERROR [VerifySnapshot-pool-0] snapshot.SnapshotReferenceUtil(301): Can't find hfile: 6da5642c1d0f4be18c5ac7efa0898774 in the real (hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8/cf/6da5642c1d0f4be18c5ac7efa0898774) or archive (hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8/cf/6da5642c1d0f4be18c5ac7efa0898774) directory for the primary table. 2023-02-11 04:00:45,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Can't find hfile: 6da5642c1d0f4be18c5ac7efa0898774 in the real (hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8/cf/6da5642c1d0f4be18c5ac7efa0898774) or archive (hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8/cf/6da5642c1d0f4be18c5ac7efa0898774) directory for the primary table. at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.concurrentVisitReferencedFiles(SnapshotReferenceUtil.java:225) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.concurrentVisitReferencedFiles(SnapshotReferenceUtil.java:187) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.verifySnapshot(SnapshotReferenceUtil.java:164) at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.restoreOrCloneSnapshot(SnapshotManager.java:827) at org.apache.hadoop.hbase.master.HMaster$14.run(HMaster.java:2493) at org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:132) at org.apache.hadoop.hbase.master.HMaster.restoreSnapshot(HMaster.java:2489) at org.apache.hadoop.hbase.master.MasterRpcServices.restoreSnapshot(MasterRpcServices.java:1469) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 04:00:45,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] ipc.CallRunner(144): callId: 358 service: MasterService methodName: RestoreSnapshot size: 115 connection: 172.31.14.131:52380 deadline: 1676088105540, exception=org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Can't find hfile: 6da5642c1d0f4be18c5ac7efa0898774 in the real (hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8/cf/6da5642c1d0f4be18c5ac7efa0898774) or archive (hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8/cf/6da5642c1d0f4be18c5ac7efa0898774) directory for the primary table. 2023-02-11 04:00:45,548 INFO [Listener at localhost/33753] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676088039685 2023-02-11 04:00:45,548 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testCorruptedSnapshot-1676088039685 2023-02-11 04:00:45,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] procedure2.ProcedureExecutor(1029): Stored pid=270, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676088039685 2023-02-11 04:00:45,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-11 04:00:45,551 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676088039685","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088045551"}]},"ts":"1676088045551"} 2023-02-11 04:00:45,552 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676088039685, state=DISABLING in hbase:meta 2023-02-11 04:00:45,555 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676088039685 to state=DISABLING 2023-02-11 04:00:45,556 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=271, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=73f6bdfd53354bbf7c9d40fd769c9e56, UNASSIGN}, {pid=272, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=a388dd856b0a069ab94fe6f6b9fb6351, UNASSIGN}, {pid=273, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=660b743875c88bf2eaa58aab9da56413, UNASSIGN}, {pid=274, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=ee86f4ea9182f5cda2013084d379724f, UNASSIGN}, {pid=275, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=393f61e585ea6ccae9181b305adcbdf8, UNASSIGN}, {pid=276, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=998536da7be614a24a93f9e7ec31dbca, UNASSIGN}] 2023-02-11 04:00:45,556 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=275, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=393f61e585ea6ccae9181b305adcbdf8, UNASSIGN 2023-02-11 04:00:45,556 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=273, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=660b743875c88bf2eaa58aab9da56413, UNASSIGN 2023-02-11 04:00:45,556 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=274, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=ee86f4ea9182f5cda2013084d379724f, UNASSIGN 2023-02-11 04:00:45,556 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=272, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=a388dd856b0a069ab94fe6f6b9fb6351, UNASSIGN 2023-02-11 04:00:45,557 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=276, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=998536da7be614a24a93f9e7ec31dbca, UNASSIGN 2023-02-11 04:00:45,557 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=275 updating hbase:meta row=393f61e585ea6ccae9181b305adcbdf8, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:45,557 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088045557"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088045557"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088045557"}]},"ts":"1676088045557"} 2023-02-11 04:00:45,557 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=273 updating hbase:meta row=660b743875c88bf2eaa58aab9da56413, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:45,557 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=274 updating hbase:meta row=ee86f4ea9182f5cda2013084d379724f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:45,557 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=276 updating hbase:meta row=998536da7be614a24a93f9e7ec31dbca, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:45,557 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088045557"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088045557"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088045557"}]},"ts":"1676088045557"} 2023-02-11 04:00:45,557 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676088045557"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088045557"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088045557"}]},"ts":"1676088045557"} 2023-02-11 04:00:45,557 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088045557"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088045557"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088045557"}]},"ts":"1676088045557"} 2023-02-11 04:00:45,557 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=272 updating hbase:meta row=a388dd856b0a069ab94fe6f6b9fb6351, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:45,558 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088045557"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088045557"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088045557"}]},"ts":"1676088045557"} 2023-02-11 04:00:45,558 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=277, ppid=275, state=RUNNABLE; CloseRegionProcedure 393f61e585ea6ccae9181b305adcbdf8, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:45,559 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=278, ppid=274, state=RUNNABLE; CloseRegionProcedure ee86f4ea9182f5cda2013084d379724f, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:45,559 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=279, ppid=276, state=RUNNABLE; CloseRegionProcedure 998536da7be614a24a93f9e7ec31dbca, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:45,559 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=280, ppid=273, state=RUNNABLE; CloseRegionProcedure 660b743875c88bf2eaa58aab9da56413, server=jenkins-hbase4.apache.org,46851,1676088013694}] 2023-02-11 04:00:45,560 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=281, ppid=272, state=RUNNABLE; CloseRegionProcedure a388dd856b0a069ab94fe6f6b9fb6351, server=jenkins-hbase4.apache.org,42385,1676088013771}] 2023-02-11 04:00:45,561 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=271, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=73f6bdfd53354bbf7c9d40fd769c9e56, UNASSIGN 2023-02-11 04:00:45,561 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=271 updating hbase:meta row=73f6bdfd53354bbf7c9d40fd769c9e56, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:45,561 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676088045561"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676088045561"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676088045561"}]},"ts":"1676088045561"} 2023-02-11 04:00:45,562 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=282, ppid=271, state=RUNNABLE; CloseRegionProcedure 73f6bdfd53354bbf7c9d40fd769c9e56, server=jenkins-hbase4.apache.org,40829,1676088013739}] 2023-02-11 04:00:45,710 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 393f61e585ea6ccae9181b305adcbdf8 2023-02-11 04:00:45,710 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 73f6bdfd53354bbf7c9d40fd769c9e56 2023-02-11 04:00:45,710 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 998536da7be614a24a93f9e7ec31dbca 2023-02-11 04:00:45,710 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 73f6bdfd53354bbf7c9d40fd769c9e56, disabling compactions & flushes 2023-02-11 04:00:45,710 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 998536da7be614a24a93f9e7ec31dbca, disabling compactions & flushes 2023-02-11 04:00:45,711 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. 2023-02-11 04:00:45,710 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 393f61e585ea6ccae9181b305adcbdf8, disabling compactions & flushes 2023-02-11 04:00:45,711 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. 2023-02-11 04:00:45,711 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8. 2023-02-11 04:00:45,711 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca. 2023-02-11 04:00:45,711 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8. 2023-02-11 04:00:45,711 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8. after waiting 0 ms 2023-02-11 04:00:45,711 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8. 2023-02-11 04:00:45,711 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. after waiting 0 ms 2023-02-11 04:00:45,711 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 393f61e585ea6ccae9181b305adcbdf8 1/1 column families, dataSize=2.16 KB heapSize=4.89 KB 2023-02-11 04:00:45,711 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca. 2023-02-11 04:00:45,711 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. 2023-02-11 04:00:45,711 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca. after waiting 0 ms 2023-02-11 04:00:45,711 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca. 2023-02-11 04:00:45,711 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 73f6bdfd53354bbf7c9d40fd769c9e56 1/1 column families, dataSize=1.70 KB heapSize=3.91 KB 2023-02-11 04:00:45,711 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 998536da7be614a24a93f9e7ec31dbca 1/1 column families, dataSize=23.29 KB heapSize=50.31 KB 2023-02-11 04:00:45,732 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c20230211df87d133b0714cc7b7f374d4775a3c1e to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/a87ff679a2f3e71d9181a67b7542122c20230211df87d133b0714cc7b7f374d4775a3c1e 2023-02-11 04:00:45,732 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230211409775d3db474960b444dd434eb6baa7 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/d41d8cd98f00b204e9800998ecf8427e20230211409775d3db474960b444dd434eb6baa7 2023-02-11 04:00:45,732 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d5202302117e7bb5d67c4e4ff4a45580bc44f8d561 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/e4da3b7fbbce2345d7772b0674a318d5202302117e7bb5d67c4e4ff4a45580bc44f8d561 2023-02-11 04:00:45,741 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8/.tmp/cf/070d38afc33a4f9f93317edb5c4499a0 2023-02-11 04:00:45,742 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56/.tmp/cf/4a3d5b407b924e65850a8c87419cfd38 2023-02-11 04:00:45,743 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=23.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca/.tmp/cf/b90c3e95bfa24f8280d697a46dd41fc7 2023-02-11 04:00:45,745 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8/.tmp/cf/070d38afc33a4f9f93317edb5c4499a0 as hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8/cf/070d38afc33a4f9f93317edb5c4499a0 2023-02-11 04:00:45,745 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56/.tmp/cf/4a3d5b407b924e65850a8c87419cfd38 as hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56/cf/4a3d5b407b924e65850a8c87419cfd38 2023-02-11 04:00:45,746 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca/.tmp/cf/b90c3e95bfa24f8280d697a46dd41fc7 as hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca/cf/b90c3e95bfa24f8280d697a46dd41fc7 2023-02-11 04:00:45,748 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56/cf/4a3d5b407b924e65850a8c87419cfd38, entries=26, sequenceid=15, filesize=9.4 K 2023-02-11 04:00:45,748 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8/cf/070d38afc33a4f9f93317edb5c4499a0, entries=33, sequenceid=15, filesize=10.6 K 2023-02-11 04:00:45,749 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca/cf/b90c3e95bfa24f8280d697a46dd41fc7, entries=356, sequenceid=15, filesize=66.2 K 2023-02-11 04:00:45,751 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.70 KB/1740, heapSize ~3.89 KB/3984, currentSize=0 B/0 for 73f6bdfd53354bbf7c9d40fd769c9e56 in 38ms, sequenceid=15, compaction requested=false 2023-02-11 04:00:45,751 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.16 KB/2209, heapSize ~4.88 KB/4992, currentSize=0 B/0 for 393f61e585ea6ccae9181b305adcbdf8 in 40ms, sequenceid=15, compaction requested=false 2023-02-11 04:00:45,751 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~23.29 KB/23848, heapSize ~50.30 KB/51504, currentSize=0 B/0 for 998536da7be614a24a93f9e7ec31dbca in 40ms, sequenceid=15, compaction requested=false 2023-02-11 04:00:45,754 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 04:00:45,754 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 04:00:45,754 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 04:00:45,755 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. 2023-02-11 04:00:45,755 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8. 2023-02-11 04:00:45,755 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 73f6bdfd53354bbf7c9d40fd769c9e56: Waiting for close lock at 1676088045710Running coprocessor pre-close hooks at 1676088045710Disabling compacts and flushes for region at 1676088045710Disabling writes for close at 1676088045711 (+1 ms)Obtaining lock to block concurrent updates at 1676088045711Preparing flush snapshotting stores in 73f6bdfd53354bbf7c9d40fd769c9e56 at 1676088045711Finished memstore snapshotting testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56., syncing WAL and waiting on mvcc, flushsize=dataSize=1740, getHeapSize=3984, getOffHeapSize=0, getCellsCount=26 at 1676088045711Flushing stores of testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56. at 1676088045713 (+2 ms)Flushing 73f6bdfd53354bbf7c9d40fd769c9e56/cf: creating writer at 1676088045714 (+1 ms)Flushing mob file 73f6bdfd53354bbf7c9d40fd769c9e56/cf: appending metadata at 1676088045718 (+4 ms)Flushing mob file 73f6bdfd53354bbf7c9d40fd769c9e56/cf: closing flushed file at 1676088045718Flushing 73f6bdfd53354bbf7c9d40fd769c9e56/cf: appending metadata at 1676088045733 (+15 ms)Flushing 73f6bdfd53354bbf7c9d40fd769c9e56/cf: closing flushed file at 1676088045733Flushing 73f6bdfd53354bbf7c9d40fd769c9e56/cf: reopening flushed file at 1676088045746 (+13 ms)Finished flush of dataSize ~1.70 KB/1740, heapSize ~3.89 KB/3984, currentSize=0 B/0 for 73f6bdfd53354bbf7c9d40fd769c9e56 in 38ms, sequenceid=15, compaction requested=false at 1676088045751 (+5 ms)Writing region close event to WAL at 1676088045752 (+1 ms)Running coprocessor post-close hooks at 1676088045754 (+2 ms)Closed at 1676088045755 (+1 ms) 2023-02-11 04:00:45,755 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 393f61e585ea6ccae9181b305adcbdf8: Waiting for close lock at 1676088045710Running coprocessor pre-close hooks at 1676088045710Disabling compacts and flushes for region at 1676088045710Disabling writes for close at 1676088045711 (+1 ms)Obtaining lock to block concurrent updates at 1676088045711Preparing flush snapshotting stores in 393f61e585ea6ccae9181b305adcbdf8 at 1676088045711Finished memstore snapshotting testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8., syncing WAL and waiting on mvcc, flushsize=dataSize=2209, getHeapSize=4992, getOffHeapSize=0, getCellsCount=33 at 1676088045711Flushing stores of testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8. at 1676088045712 (+1 ms)Flushing 393f61e585ea6ccae9181b305adcbdf8/cf: creating writer at 1676088045713 (+1 ms)Flushing mob file 393f61e585ea6ccae9181b305adcbdf8/cf: appending metadata at 1676088045718 (+5 ms)Flushing mob file 393f61e585ea6ccae9181b305adcbdf8/cf: closing flushed file at 1676088045718Flushing 393f61e585ea6ccae9181b305adcbdf8/cf: appending metadata at 1676088045733 (+15 ms)Flushing 393f61e585ea6ccae9181b305adcbdf8/cf: closing flushed file at 1676088045733Flushing 393f61e585ea6ccae9181b305adcbdf8/cf: reopening flushed file at 1676088045745 (+12 ms)Finished flush of dataSize ~2.16 KB/2209, heapSize ~4.88 KB/4992, currentSize=0 B/0 for 393f61e585ea6ccae9181b305adcbdf8 in 40ms, sequenceid=15, compaction requested=false at 1676088045751 (+6 ms)Writing region close event to WAL at 1676088045752 (+1 ms)Running coprocessor post-close hooks at 1676088045755 (+3 ms)Closed at 1676088045755 2023-02-11 04:00:45,755 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca. 2023-02-11 04:00:45,755 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 998536da7be614a24a93f9e7ec31dbca: Waiting for close lock at 1676088045710Running coprocessor pre-close hooks at 1676088045710Disabling compacts and flushes for region at 1676088045710Disabling writes for close at 1676088045711 (+1 ms)Obtaining lock to block concurrent updates at 1676088045711Preparing flush snapshotting stores in 998536da7be614a24a93f9e7ec31dbca at 1676088045711Finished memstore snapshotting testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca., syncing WAL and waiting on mvcc, flushsize=dataSize=23848, getHeapSize=51504, getOffHeapSize=0, getCellsCount=356 at 1676088045711Flushing stores of testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca. at 1676088045714 (+3 ms)Flushing 998536da7be614a24a93f9e7ec31dbca/cf: creating writer at 1676088045714Flushing mob file 998536da7be614a24a93f9e7ec31dbca/cf: appending metadata at 1676088045720 (+6 ms)Flushing mob file 998536da7be614a24a93f9e7ec31dbca/cf: closing flushed file at 1676088045720Flushing 998536da7be614a24a93f9e7ec31dbca/cf: appending metadata at 1676088045733 (+13 ms)Flushing 998536da7be614a24a93f9e7ec31dbca/cf: closing flushed file at 1676088045733Flushing 998536da7be614a24a93f9e7ec31dbca/cf: reopening flushed file at 1676088045746 (+13 ms)Finished flush of dataSize ~23.29 KB/23848, heapSize ~50.30 KB/51504, currentSize=0 B/0 for 998536da7be614a24a93f9e7ec31dbca in 40ms, sequenceid=15, compaction requested=false at 1676088045752 (+6 ms)Writing region close event to WAL at 1676088045752Running coprocessor post-close hooks at 1676088045755 (+3 ms)Closed at 1676088045755 2023-02-11 04:00:45,756 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 73f6bdfd53354bbf7c9d40fd769c9e56 2023-02-11 04:00:45,756 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ee86f4ea9182f5cda2013084d379724f 2023-02-11 04:00:45,756 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing ee86f4ea9182f5cda2013084d379724f, disabling compactions & flushes 2023-02-11 04:00:45,756 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f. 2023-02-11 04:00:45,756 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f. 2023-02-11 04:00:45,756 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f. after waiting 0 ms 2023-02-11 04:00:45,756 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f. 2023-02-11 04:00:45,756 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=271 updating hbase:meta row=73f6bdfd53354bbf7c9d40fd769c9e56, regionState=CLOSED 2023-02-11 04:00:45,756 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing ee86f4ea9182f5cda2013084d379724f 1/1 column families, dataSize=2.03 KB heapSize=4.61 KB 2023-02-11 04:00:45,756 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676088045756"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088045756"}]},"ts":"1676088045756"} 2023-02-11 04:00:45,756 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 393f61e585ea6ccae9181b305adcbdf8 2023-02-11 04:00:45,756 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 660b743875c88bf2eaa58aab9da56413 2023-02-11 04:00:45,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 660b743875c88bf2eaa58aab9da56413, disabling compactions & flushes 2023-02-11 04:00:45,757 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413. 2023-02-11 04:00:45,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413. 2023-02-11 04:00:45,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413. after waiting 0 ms 2023-02-11 04:00:45,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413. 2023-02-11 04:00:45,757 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 660b743875c88bf2eaa58aab9da56413 1/1 column families, dataSize=1.70 KB heapSize=3.91 KB 2023-02-11 04:00:45,759 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=275 updating hbase:meta row=393f61e585ea6ccae9181b305adcbdf8, regionState=CLOSED 2023-02-11 04:00:45,759 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088045759"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088045759"}]},"ts":"1676088045759"} 2023-02-11 04:00:45,759 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 998536da7be614a24a93f9e7ec31dbca 2023-02-11 04:00:45,759 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a388dd856b0a069ab94fe6f6b9fb6351 2023-02-11 04:00:45,759 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing a388dd856b0a069ab94fe6f6b9fb6351, disabling compactions & flushes 2023-02-11 04:00:45,759 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351. 2023-02-11 04:00:45,759 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351. 2023-02-11 04:00:45,759 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351. after waiting 0 ms 2023-02-11 04:00:45,759 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351. 2023-02-11 04:00:45,760 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing a388dd856b0a069ab94fe6f6b9fb6351 1/1 column families, dataSize=1.83 KB heapSize=4.19 KB 2023-02-11 04:00:45,760 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=276 updating hbase:meta row=998536da7be614a24a93f9e7ec31dbca, regionState=CLOSED 2023-02-11 04:00:45,760 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676088045760"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088045760"}]},"ts":"1676088045760"} 2023-02-11 04:00:45,762 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=282, resume processing ppid=271 2023-02-11 04:00:45,762 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=282, ppid=271, state=SUCCESS; CloseRegionProcedure 73f6bdfd53354bbf7c9d40fd769c9e56, server=jenkins-hbase4.apache.org,40829,1676088013739 in 197 msec 2023-02-11 04:00:45,763 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=277, resume processing ppid=275 2023-02-11 04:00:45,763 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=271, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=73f6bdfd53354bbf7c9d40fd769c9e56, UNASSIGN in 207 msec 2023-02-11 04:00:45,763 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=277, ppid=275, state=SUCCESS; CloseRegionProcedure 393f61e585ea6ccae9181b305adcbdf8, server=jenkins-hbase4.apache.org,46851,1676088013694 in 203 msec 2023-02-11 04:00:45,764 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=279, resume processing ppid=276 2023-02-11 04:00:45,764 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=279, ppid=276, state=SUCCESS; CloseRegionProcedure 998536da7be614a24a93f9e7ec31dbca, server=jenkins-hbase4.apache.org,42385,1676088013771 in 203 msec 2023-02-11 04:00:45,765 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=275, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=393f61e585ea6ccae9181b305adcbdf8, UNASSIGN in 208 msec 2023-02-11 04:00:45,765 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=276, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=998536da7be614a24a93f9e7ec31dbca, UNASSIGN in 209 msec 2023-02-11 04:00:45,782 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf32023021177892238eaeb4dcaa0030931ac10ab46 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021177892238eaeb4dcaa0030931ac10ab46 2023-02-11 04:00:45,782 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c20230211ef6d4bf876bf4a249eac9306f73c7e65 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/c81e728d9d4c2f636f067f89cc14862c20230211ef6d4bf876bf4a249eac9306f73c7e65 2023-02-11 04:00:45,782 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b20230211dda1066663d64a48bbfa3d662cd516df to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/c4ca4238a0b923820dcc509a6f75849b20230211dda1066663d64a48bbfa3d662cd516df 2023-02-11 04:00:45,792 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351/.tmp/cf/5edd17b1fddf46ffa6bd3e1809ee3cfe 2023-02-11 04:00:45,792 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413/.tmp/cf/b18d9f5b148a48578519a37094d1daf4 2023-02-11 04:00:45,793 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f/.tmp/cf/bf9e69da156e4fee8fcae0a5ce197edb 2023-02-11 04:00:45,796 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351/.tmp/cf/5edd17b1fddf46ffa6bd3e1809ee3cfe as hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351/cf/5edd17b1fddf46ffa6bd3e1809ee3cfe 2023-02-11 04:00:45,796 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413/.tmp/cf/b18d9f5b148a48578519a37094d1daf4 as hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413/cf/b18d9f5b148a48578519a37094d1daf4 2023-02-11 04:00:45,797 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f/.tmp/cf/bf9e69da156e4fee8fcae0a5ce197edb as hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f/cf/bf9e69da156e4fee8fcae0a5ce197edb 2023-02-11 04:00:45,799 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351/cf/5edd17b1fddf46ffa6bd3e1809ee3cfe, entries=28, sequenceid=15, filesize=9.7 K 2023-02-11 04:00:45,799 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413/cf/b18d9f5b148a48578519a37094d1daf4, entries=26, sequenceid=15, filesize=9.4 K 2023-02-11 04:00:45,800 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.83 KB/1874, heapSize ~4.17 KB/4272, currentSize=0 B/0 for a388dd856b0a069ab94fe6f6b9fb6351 in 41ms, sequenceid=15, compaction requested=false 2023-02-11 04:00:45,800 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.70 KB/1740, heapSize ~3.89 KB/3984, currentSize=0 B/0 for 660b743875c88bf2eaa58aab9da56413 in 43ms, sequenceid=15, compaction requested=false 2023-02-11 04:00:45,800 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f/cf/bf9e69da156e4fee8fcae0a5ce197edb, entries=31, sequenceid=15, filesize=10.2 K 2023-02-11 04:00:45,801 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for ee86f4ea9182f5cda2013084d379724f in 45ms, sequenceid=15, compaction requested=false 2023-02-11 04:00:45,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-11 04:00:45,803 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 04:00:45,803 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 04:00:45,803 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413. 2023-02-11 04:00:45,803 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 660b743875c88bf2eaa58aab9da56413: Waiting for close lock at 1676088045757Running coprocessor pre-close hooks at 1676088045757Disabling compacts and flushes for region at 1676088045757Disabling writes for close at 1676088045757Obtaining lock to block concurrent updates at 1676088045757Preparing flush snapshotting stores in 660b743875c88bf2eaa58aab9da56413 at 1676088045757Finished memstore snapshotting testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413., syncing WAL and waiting on mvcc, flushsize=dataSize=1740, getHeapSize=3984, getOffHeapSize=0, getCellsCount=26 at 1676088045758 (+1 ms)Flushing stores of testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413. at 1676088045758Flushing 660b743875c88bf2eaa58aab9da56413/cf: creating writer at 1676088045759 (+1 ms)Flushing mob file 660b743875c88bf2eaa58aab9da56413/cf: appending metadata at 1676088045766 (+7 ms)Flushing mob file 660b743875c88bf2eaa58aab9da56413/cf: closing flushed file at 1676088045766Flushing 660b743875c88bf2eaa58aab9da56413/cf: appending metadata at 1676088045782 (+16 ms)Flushing 660b743875c88bf2eaa58aab9da56413/cf: closing flushed file at 1676088045782Flushing 660b743875c88bf2eaa58aab9da56413/cf: reopening flushed file at 1676088045797 (+15 ms)Finished flush of dataSize ~1.70 KB/1740, heapSize ~3.89 KB/3984, currentSize=0 B/0 for 660b743875c88bf2eaa58aab9da56413 in 43ms, sequenceid=15, compaction requested=false at 1676088045800 (+3 ms)Writing region close event to WAL at 1676088045800Running coprocessor post-close hooks at 1676088045803 (+3 ms)Closed at 1676088045803 2023-02-11 04:00:45,804 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 04:00:45,804 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351. 2023-02-11 04:00:45,804 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for a388dd856b0a069ab94fe6f6b9fb6351: Waiting for close lock at 1676088045759Running coprocessor pre-close hooks at 1676088045759Disabling compacts and flushes for region at 1676088045759Disabling writes for close at 1676088045759Obtaining lock to block concurrent updates at 1676088045760 (+1 ms)Preparing flush snapshotting stores in a388dd856b0a069ab94fe6f6b9fb6351 at 1676088045760Finished memstore snapshotting testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351., syncing WAL and waiting on mvcc, flushsize=dataSize=1874, getHeapSize=4272, getOffHeapSize=0, getCellsCount=28 at 1676088045760Flushing stores of testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351. at 1676088045760Flushing a388dd856b0a069ab94fe6f6b9fb6351/cf: creating writer at 1676088045761 (+1 ms)Flushing mob file a388dd856b0a069ab94fe6f6b9fb6351/cf: appending metadata at 1676088045767 (+6 ms)Flushing mob file a388dd856b0a069ab94fe6f6b9fb6351/cf: closing flushed file at 1676088045767Flushing a388dd856b0a069ab94fe6f6b9fb6351/cf: appending metadata at 1676088045783 (+16 ms)Flushing a388dd856b0a069ab94fe6f6b9fb6351/cf: closing flushed file at 1676088045783Flushing a388dd856b0a069ab94fe6f6b9fb6351/cf: reopening flushed file at 1676088045796 (+13 ms)Finished flush of dataSize ~1.83 KB/1874, heapSize ~4.17 KB/4272, currentSize=0 B/0 for a388dd856b0a069ab94fe6f6b9fb6351 in 41ms, sequenceid=15, compaction requested=false at 1676088045800 (+4 ms)Writing region close event to WAL at 1676088045800Running coprocessor post-close hooks at 1676088045804 (+4 ms)Closed at 1676088045804 2023-02-11 04:00:45,804 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f. 2023-02-11 04:00:45,804 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for ee86f4ea9182f5cda2013084d379724f: Waiting for close lock at 1676088045756Running coprocessor pre-close hooks at 1676088045756Disabling compacts and flushes for region at 1676088045756Disabling writes for close at 1676088045756Obtaining lock to block concurrent updates at 1676088045756Preparing flush snapshotting stores in ee86f4ea9182f5cda2013084d379724f at 1676088045756Finished memstore snapshotting testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f., syncing WAL and waiting on mvcc, flushsize=dataSize=2075, getHeapSize=4704, getOffHeapSize=0, getCellsCount=31 at 1676088045756Flushing stores of testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f. at 1676088045757 (+1 ms)Flushing ee86f4ea9182f5cda2013084d379724f/cf: creating writer at 1676088045758 (+1 ms)Flushing mob file ee86f4ea9182f5cda2013084d379724f/cf: appending metadata at 1676088045764 (+6 ms)Flushing mob file ee86f4ea9182f5cda2013084d379724f/cf: closing flushed file at 1676088045764Flushing ee86f4ea9182f5cda2013084d379724f/cf: appending metadata at 1676088045782 (+18 ms)Flushing ee86f4ea9182f5cda2013084d379724f/cf: closing flushed file at 1676088045782Flushing ee86f4ea9182f5cda2013084d379724f/cf: reopening flushed file at 1676088045797 (+15 ms)Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for ee86f4ea9182f5cda2013084d379724f in 45ms, sequenceid=15, compaction requested=false at 1676088045801 (+4 ms)Writing region close event to WAL at 1676088045801Running coprocessor post-close hooks at 1676088045804 (+3 ms)Closed at 1676088045804 2023-02-11 04:00:45,804 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 660b743875c88bf2eaa58aab9da56413 2023-02-11 04:00:45,805 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=273 updating hbase:meta row=660b743875c88bf2eaa58aab9da56413, regionState=CLOSED 2023-02-11 04:00:45,805 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088045805"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088045805"}]},"ts":"1676088045805"} 2023-02-11 04:00:45,805 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a388dd856b0a069ab94fe6f6b9fb6351 2023-02-11 04:00:45,805 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=272 updating hbase:meta row=a388dd856b0a069ab94fe6f6b9fb6351, regionState=CLOSED 2023-02-11 04:00:45,805 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088045805"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088045805"}]},"ts":"1676088045805"} 2023-02-11 04:00:45,806 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ee86f4ea9182f5cda2013084d379724f 2023-02-11 04:00:45,806 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=274 updating hbase:meta row=ee86f4ea9182f5cda2013084d379724f, regionState=CLOSED 2023-02-11 04:00:45,806 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676088045806"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676088045806"}]},"ts":"1676088045806"} 2023-02-11 04:00:45,807 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=280, resume processing ppid=273 2023-02-11 04:00:45,807 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=280, ppid=273, state=SUCCESS; CloseRegionProcedure 660b743875c88bf2eaa58aab9da56413, server=jenkins-hbase4.apache.org,46851,1676088013694 in 247 msec 2023-02-11 04:00:45,808 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=281, resume processing ppid=272 2023-02-11 04:00:45,808 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=281, ppid=272, state=SUCCESS; CloseRegionProcedure a388dd856b0a069ab94fe6f6b9fb6351, server=jenkins-hbase4.apache.org,42385,1676088013771 in 246 msec 2023-02-11 04:00:45,808 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=273, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=660b743875c88bf2eaa58aab9da56413, UNASSIGN in 252 msec 2023-02-11 04:00:45,809 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=278, resume processing ppid=274 2023-02-11 04:00:45,809 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=272, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=a388dd856b0a069ab94fe6f6b9fb6351, UNASSIGN in 253 msec 2023-02-11 04:00:45,809 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=278, ppid=274, state=SUCCESS; CloseRegionProcedure ee86f4ea9182f5cda2013084d379724f, server=jenkins-hbase4.apache.org,40829,1676088013739 in 249 msec 2023-02-11 04:00:45,810 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=274, resume processing ppid=270 2023-02-11 04:00:45,810 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=274, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676088039685, region=ee86f4ea9182f5cda2013084d379724f, UNASSIGN in 254 msec 2023-02-11 04:00:45,810 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676088039685","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676088045810"}]},"ts":"1676088045810"} 2023-02-11 04:00:45,811 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676088039685, state=DISABLED in hbase:meta 2023-02-11 04:00:45,812 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676088039685 to state=DISABLED 2023-02-11 04:00:45,814 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=270, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676088039685 in 264 msec 2023-02-11 04:00:45,993 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-11 04:00:46,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-11 04:00:46,303 INFO [Listener at localhost/33753] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676088039685, procId: 270 completed 2023-02-11 04:00:46,304 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.HMaster$5(2215): Client=jenkins//172.31.14.131 delete testCorruptedSnapshot-1676088039685 2023-02-11 04:00:46,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] procedure2.ProcedureExecutor(1029): Stored pid=283, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testCorruptedSnapshot-1676088039685 2023-02-11 04:00:46,306 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=283, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676088039685 2023-02-11 04:00:46,306 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=283, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676088039685 2023-02-11 04:00:46,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=283 2023-02-11 04:00:46,313 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56 2023-02-11 04:00:46,313 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca 2023-02-11 04:00:46,313 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f 2023-02-11 04:00:46,313 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8 2023-02-11 04:00:46,313 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413 2023-02-11 04:00:46,313 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351 2023-02-11 04:00:46,315 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413/cf, FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413/recovered.edits] 2023-02-11 04:00:46,315 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca/cf, FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca/recovered.edits] 2023-02-11 04:00:46,316 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56/cf, FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56/recovered.edits] 2023-02-11 04:00:46,316 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f/cf, FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f/recovered.edits] 2023-02-11 04:00:46,316 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8/cf, FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8/recovered.edits] 2023-02-11 04:00:46,316 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351/cf, FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351/recovered.edits] 2023-02-11 04:00:46,320 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8/cf/070d38afc33a4f9f93317edb5c4499a0 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8/cf/070d38afc33a4f9f93317edb5c4499a0 2023-02-11 04:00:46,321 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca/cf/b90c3e95bfa24f8280d697a46dd41fc7 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca/cf/b90c3e95bfa24f8280d697a46dd41fc7 2023-02-11 04:00:46,321 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351/cf/344f39904f8845f7bb49a0168e680fc3 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351/cf/344f39904f8845f7bb49a0168e680fc3 2023-02-11 04:00:46,321 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f/cf/7b45a4d0a7ec4949ae74442957ffa428 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f/cf/7b45a4d0a7ec4949ae74442957ffa428 2023-02-11 04:00:46,321 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413/cf/60aa41415b5e455e9c66d4ddf3b44e2c to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413/cf/60aa41415b5e455e9c66d4ddf3b44e2c 2023-02-11 04:00:46,322 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56/cf/2bb70252cfa54ef988c157f9aa816002 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56/cf/2bb70252cfa54ef988c157f9aa816002 2023-02-11 04:00:46,323 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413/cf/b18d9f5b148a48578519a37094d1daf4 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413/cf/b18d9f5b148a48578519a37094d1daf4 2023-02-11 04:00:46,323 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351/cf/5edd17b1fddf46ffa6bd3e1809ee3cfe to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351/cf/5edd17b1fddf46ffa6bd3e1809ee3cfe 2023-02-11 04:00:46,323 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f/cf/bf9e69da156e4fee8fcae0a5ce197edb to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f/cf/bf9e69da156e4fee8fcae0a5ce197edb 2023-02-11 04:00:46,323 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca/cf/cc67fb5d7fb94be88db8bea101177d4b to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca/cf/cc67fb5d7fb94be88db8bea101177d4b 2023-02-11 04:00:46,324 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8/recovered.edits/18.seqid to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8/recovered.edits/18.seqid 2023-02-11 04:00:46,324 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56/cf/4a3d5b407b924e65850a8c87419cfd38 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56/cf/4a3d5b407b924e65850a8c87419cfd38 2023-02-11 04:00:46,325 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/393f61e585ea6ccae9181b305adcbdf8 2023-02-11 04:00:46,327 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f/recovered.edits/18.seqid to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f/recovered.edits/18.seqid 2023-02-11 04:00:46,327 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351/recovered.edits/18.seqid to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351/recovered.edits/18.seqid 2023-02-11 04:00:46,327 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca/recovered.edits/18.seqid to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca/recovered.edits/18.seqid 2023-02-11 04:00:46,327 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413/recovered.edits/18.seqid to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413/recovered.edits/18.seqid 2023-02-11 04:00:46,327 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/998536da7be614a24a93f9e7ec31dbca 2023-02-11 04:00:46,327 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/660b743875c88bf2eaa58aab9da56413 2023-02-11 04:00:46,328 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56/recovered.edits/18.seqid to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56/recovered.edits/18.seqid 2023-02-11 04:00:46,328 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/ee86f4ea9182f5cda2013084d379724f 2023-02-11 04:00:46,328 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/a388dd856b0a069ab94fe6f6b9fb6351 2023-02-11 04:00:46,328 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/.tmp/data/default/testCorruptedSnapshot-1676088039685/73f6bdfd53354bbf7c9d40fd769c9e56 2023-02-11 04:00:46,328 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived testCorruptedSnapshot-1676088039685 regions 2023-02-11 04:00:46,328 DEBUG [PEWorker-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4 2023-02-11 04:00:46,329 DEBUG [PEWorker-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf] 2023-02-11 04:00:46,332 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/a87ff679a2f3e71d9181a67b7542122c20230211428491bd10e34f1cbf8b6789df0ee1c1 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/a87ff679a2f3e71d9181a67b7542122c20230211428491bd10e34f1cbf8b6789df0ee1c1 2023-02-11 04:00:46,333 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/a87ff679a2f3e71d9181a67b7542122c20230211df87d133b0714cc7b7f374d4775a3c1e to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/a87ff679a2f3e71d9181a67b7542122c20230211df87d133b0714cc7b7f374d4775a3c1e 2023-02-11 04:00:46,334 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/c4ca4238a0b923820dcc509a6f75849b2023021131b6523e847f4a289e9b074bbe60ec35 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/c4ca4238a0b923820dcc509a6f75849b2023021131b6523e847f4a289e9b074bbe60ec35 2023-02-11 04:00:46,335 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/c4ca4238a0b923820dcc509a6f75849b20230211dda1066663d64a48bbfa3d662cd516df to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/c4ca4238a0b923820dcc509a6f75849b20230211dda1066663d64a48bbfa3d662cd516df 2023-02-11 04:00:46,336 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/c81e728d9d4c2f636f067f89cc14862c2023021159ff96bbc053428e975bf9c29ce19ced to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/c81e728d9d4c2f636f067f89cc14862c2023021159ff96bbc053428e975bf9c29ce19ced 2023-02-11 04:00:46,337 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/c81e728d9d4c2f636f067f89cc14862c20230211ef6d4bf876bf4a249eac9306f73c7e65 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/c81e728d9d4c2f636f067f89cc14862c20230211ef6d4bf876bf4a249eac9306f73c7e65 2023-02-11 04:00:46,338 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/d41d8cd98f00b204e9800998ecf8427e20230211409775d3db474960b444dd434eb6baa7 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/d41d8cd98f00b204e9800998ecf8427e20230211409775d3db474960b444dd434eb6baa7 2023-02-11 04:00:46,338 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/d41d8cd98f00b204e9800998ecf8427e20230211d7f6567412ca47efb6bd3cbf605601b4 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/d41d8cd98f00b204e9800998ecf8427e20230211d7f6567412ca47efb6bd3cbf605601b4 2023-02-11 04:00:46,339 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/e4da3b7fbbce2345d7772b0674a318d5202302117e7bb5d67c4e4ff4a45580bc44f8d561 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/e4da3b7fbbce2345d7772b0674a318d5202302117e7bb5d67c4e4ff4a45580bc44f8d561 2023-02-11 04:00:46,340 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/e4da3b7fbbce2345d7772b0674a318d520230211a94c568527944d96aba6533e72345c0f to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/e4da3b7fbbce2345d7772b0674a318d520230211a94c568527944d96aba6533e72345c0f 2023-02-11 04:00:46,341 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021177892238eaeb4dcaa0030931ac10ab46 to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021177892238eaeb4dcaa0030931ac10ab46 2023-02-11 04:00:46,342 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211da45a7df0bf840a48ed16c26a4f5c52c to hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/archive/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211da45a7df0bf840a48ed16c26a4f5c52c 2023-02-11 04:00:46,343 DEBUG [PEWorker-3] backup.HFileArchiver(596): Deleted hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/mobdir/data/default/testCorruptedSnapshot-1676088039685/d3ba0293077c512468941f2c36bb61a4 2023-02-11 04:00:46,344 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=283, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676088039685 2023-02-11 04:00:46,346 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testCorruptedSnapshot-1676088039685 from hbase:meta 2023-02-11 04:00:46,348 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'testCorruptedSnapshot-1676088039685' descriptor. 2023-02-11 04:00:46,349 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=283, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676088039685 2023-02-11 04:00:46,349 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'testCorruptedSnapshot-1676088039685' from region states. 2023-02-11 04:00:46,349 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088046349"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:46,349 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088046349"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:46,349 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088046349"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:46,349 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088046349"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:46,349 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088046349"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:46,349 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676088046349"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:46,351 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-11 04:00:46,351 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 73f6bdfd53354bbf7c9d40fd769c9e56, NAME => 'testCorruptedSnapshot-1676088039685,,1676088039687.73f6bdfd53354bbf7c9d40fd769c9e56.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => a388dd856b0a069ab94fe6f6b9fb6351, NAME => 'testCorruptedSnapshot-1676088039685,1,1676088039687.a388dd856b0a069ab94fe6f6b9fb6351.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 660b743875c88bf2eaa58aab9da56413, NAME => 'testCorruptedSnapshot-1676088039685,2,1676088039687.660b743875c88bf2eaa58aab9da56413.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => ee86f4ea9182f5cda2013084d379724f, NAME => 'testCorruptedSnapshot-1676088039685,3,1676088039687.ee86f4ea9182f5cda2013084d379724f.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 393f61e585ea6ccae9181b305adcbdf8, NAME => 'testCorruptedSnapshot-1676088039685,4,1676088039687.393f61e585ea6ccae9181b305adcbdf8.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 998536da7be614a24a93f9e7ec31dbca, NAME => 'testCorruptedSnapshot-1676088039685,5,1676088039687.998536da7be614a24a93f9e7ec31dbca.', STARTKEY => '5', ENDKEY => ''}] 2023-02-11 04:00:46,351 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'testCorruptedSnapshot-1676088039685' as deleted. 2023-02-11 04:00:46,351 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676088039685","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676088046351"}]},"ts":"9223372036854775807"} 2023-02-11 04:00:46,352 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table testCorruptedSnapshot-1676088039685 state from META 2023-02-11 04:00:46,353 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=283, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676088039685 2023-02-11 04:00:46,354 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=283, state=SUCCESS; DeleteTableProcedure table=testCorruptedSnapshot-1676088039685 in 49 msec 2023-02-11 04:00:46,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(1227): Checking to see if procedure is done pid=283 2023-02-11 04:00:46,558 INFO [Listener at localhost/33753] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testCorruptedSnapshot-1676088039685, procId: 283 completed 2023-02-11 04:00:46,563 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "emptySnaptb-1676088039685" 2023-02-11 04:00:46,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(333): Deleting snapshot: emptySnaptb-1676088039685 2023-02-11 04:00:46,565 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "snaptb0-1676088039685" 2023-02-11 04:00:46,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41713] snapshot.SnapshotManager(333): Deleting snapshot: snaptb0-1676088039685 2023-02-11 04:00:46,585 INFO [Listener at localhost/33753] hbase.ResourceChecker(175): after: client.TestMobRestoreSnapshotFromClientSimple#testCorruptedSnapshot Thread=420 (was 424), OpenFileDescriptor=802 (was 827), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=229 (was 235), ProcessCount=169 (was 169), AvailableMemoryMB=2700 (was 2748) 2023-02-11 04:00:46,586 INFO [Listener at localhost/33753] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-02-11 04:00:46,586 INFO [Listener at localhost/33753] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-02-11 04:00:46,587 DEBUG [Listener at localhost/33753] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0da5f7e5 to 127.0.0.1:63425 2023-02-11 04:00:46,587 DEBUG [Listener at localhost/33753] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:00:46,587 DEBUG [Listener at localhost/33753] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-11 04:00:46,587 DEBUG [Listener at localhost/33753] util.JVMClusterUtil(257): Found active master hash=820049158, stopped=false 2023-02-11 04:00:46,587 INFO [Listener at localhost/33753] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,41713,1676088012566 2023-02-11 04:00:46,590 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): regionserver:40829-0x1004189558d0002, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-11 04:00:46,590 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): regionserver:42385-0x1004189558d0003, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-11 04:00:46,590 INFO [Listener at localhost/33753] procedure2.ProcedureExecutor(629): Stopping 2023-02-11 04:00:46,590 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): regionserver:46851-0x1004189558d0001, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-11 04:00:46,590 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-11 04:00:46,591 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 04:00:46,591 DEBUG [Listener at localhost/33753] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1fbe9909 to 127.0.0.1:63425 2023-02-11 04:00:46,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42385-0x1004189558d0003, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 04:00:46,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40829-0x1004189558d0002, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 04:00:46,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46851-0x1004189558d0001, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 04:00:46,591 DEBUG [Listener at localhost/33753] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:00:46,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 04:00:46,591 INFO [Listener at localhost/33753] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,46851,1676088013694' ***** 2023-02-11 04:00:46,592 INFO [Listener at localhost/33753] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-11 04:00:46,592 INFO [Listener at localhost/33753] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,40829,1676088013739' ***** 2023-02-11 04:00:46,592 INFO [Listener at localhost/33753] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-11 04:00:46,592 INFO [RS:0;jenkins-hbase4:46851] regionserver.HeapMemoryManager(220): Stopping 2023-02-11 04:00:46,592 INFO [RS:1;jenkins-hbase4:40829] regionserver.HeapMemoryManager(220): Stopping 2023-02-11 04:00:46,592 INFO [Listener at localhost/33753] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,42385,1676088013771' ***** 2023-02-11 04:00:46,592 INFO [Listener at localhost/33753] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-11 04:00:46,592 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-11 04:00:46,593 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-11 04:00:46,593 INFO [RS:2;jenkins-hbase4:42385] regionserver.HeapMemoryManager(220): Stopping 2023-02-11 04:00:46,592 INFO [RS:0;jenkins-hbase4:46851] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-11 04:00:46,593 INFO [RS:2;jenkins-hbase4:42385] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-11 04:00:46,592 INFO [RS:1;jenkins-hbase4:40829] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-11 04:00:46,593 INFO [RS:2;jenkins-hbase4:42385] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-11 04:00:46,593 INFO [RS:0;jenkins-hbase4:46851] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-11 04:00:46,593 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-11 04:00:46,593 INFO [RS:0;jenkins-hbase4:46851] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:46,593 INFO [RS:2;jenkins-hbase4:42385] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:46,593 DEBUG [RS:0;jenkins-hbase4:46851] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0047547e to 127.0.0.1:63425 2023-02-11 04:00:46,593 DEBUG [RS:2;jenkins-hbase4:42385] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x139497cc to 127.0.0.1:63425 2023-02-11 04:00:46,593 INFO [RS:1;jenkins-hbase4:40829] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-11 04:00:46,593 DEBUG [RS:2;jenkins-hbase4:42385] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:00:46,593 DEBUG [RS:0;jenkins-hbase4:46851] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:00:46,593 INFO [RS:1;jenkins-hbase4:40829] regionserver.HRegionServer(3304): Received CLOSE for e892275aa04716c37d873e5baba493e7 2023-02-11 04:00:46,593 INFO [RS:0;jenkins-hbase4:46851] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,46851,1676088013694; all regions closed. 2023-02-11 04:00:46,593 INFO [RS:2;jenkins-hbase4:42385] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,42385,1676088013771; all regions closed. 2023-02-11 04:00:46,594 INFO [RS:1;jenkins-hbase4:40829] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:46,594 DEBUG [RS:1;jenkins-hbase4:40829] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x748d25ce to 127.0.0.1:63425 2023-02-11 04:00:46,594 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e892275aa04716c37d873e5baba493e7, disabling compactions & flushes 2023-02-11 04:00:46,594 DEBUG [RS:1;jenkins-hbase4:40829] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:00:46,594 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676088015613.e892275aa04716c37d873e5baba493e7. 2023-02-11 04:00:46,594 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676088015613.e892275aa04716c37d873e5baba493e7. 2023-02-11 04:00:46,594 INFO [RS:1;jenkins-hbase4:40829] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-11 04:00:46,594 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676088015613.e892275aa04716c37d873e5baba493e7. after waiting 0 ms 2023-02-11 04:00:46,594 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676088015613.e892275aa04716c37d873e5baba493e7. 2023-02-11 04:00:46,594 INFO [RS:1;jenkins-hbase4:40829] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-11 04:00:46,594 INFO [RS:1;jenkins-hbase4:40829] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-11 04:00:46,594 INFO [RS:1;jenkins-hbase4:40829] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-11 04:00:46,594 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing e892275aa04716c37d873e5baba493e7 1/1 column families, dataSize=78 B heapSize=488 B 2023-02-11 04:00:46,599 INFO [RS:1;jenkins-hbase4:40829] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-02-11 04:00:46,599 DEBUG [RS:1;jenkins-hbase4:40829] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, e892275aa04716c37d873e5baba493e7=hbase:namespace,,1676088015613.e892275aa04716c37d873e5baba493e7.} 2023-02-11 04:00:46,599 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-11 04:00:46,600 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-11 04:00:46,600 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-11 04:00:46,600 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-11 04:00:46,600 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-11 04:00:46,601 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=137.71 KB heapSize=212.48 KB 2023-02-11 04:00:46,602 DEBUG [RS:1;jenkins-hbase4:40829] regionserver.HRegionServer(1505): Waiting on 1588230740, e892275aa04716c37d873e5baba493e7 2023-02-11 04:00:46,608 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/WALs/jenkins-hbase4.apache.org,42385,1676088013771/jenkins-hbase4.apache.org%2C42385%2C1676088013771.1676088015249 not finished, retry = 0 2023-02-11 04:00:46,609 DEBUG [RS:0;jenkins-hbase4:46851] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/oldWALs 2023-02-11 04:00:46,609 INFO [RS:0;jenkins-hbase4:46851] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C46851%2C1676088013694:(num 1676088015249) 2023-02-11 04:00:46,609 DEBUG [RS:0;jenkins-hbase4:46851] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:00:46,609 INFO [RS:0;jenkins-hbase4:46851] regionserver.LeaseManager(133): Closed leases 2023-02-11 04:00:46,609 INFO [RS:0;jenkins-hbase4:46851] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-11 04:00:46,609 INFO [RS:0;jenkins-hbase4:46851] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-11 04:00:46,609 INFO [RS:0;jenkins-hbase4:46851] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-11 04:00:46,609 INFO [RS:0;jenkins-hbase4:46851] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-11 04:00:46,609 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-11 04:00:46,610 INFO [RS:0;jenkins-hbase4:46851] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:46851 2023-02-11 04:00:46,618 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-11 04:00:46,620 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/namespace/e892275aa04716c37d873e5baba493e7/.tmp/info/4e909ff1aa4a4faebb33089cb3934aaf 2023-02-11 04:00:46,623 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-11 04:00:46,623 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-11 04:00:46,624 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=130.94 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/meta/1588230740/.tmp/info/330ed09142de4da98d3f2026b5dce8f8 2023-02-11 04:00:46,624 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/namespace/e892275aa04716c37d873e5baba493e7/.tmp/info/4e909ff1aa4a4faebb33089cb3934aaf as hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/namespace/e892275aa04716c37d873e5baba493e7/info/4e909ff1aa4a4faebb33089cb3934aaf 2023-02-11 04:00:46,627 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/namespace/e892275aa04716c37d873e5baba493e7/info/4e909ff1aa4a4faebb33089cb3934aaf, entries=2, sequenceid=6, filesize=4.8 K 2023-02-11 04:00:46,627 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 330ed09142de4da98d3f2026b5dce8f8 2023-02-11 04:00:46,628 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for e892275aa04716c37d873e5baba493e7 in 34ms, sequenceid=6, compaction requested=false 2023-02-11 04:00:46,632 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/namespace/e892275aa04716c37d873e5baba493e7/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-02-11 04:00:46,633 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676088015613.e892275aa04716c37d873e5baba493e7. 2023-02-11 04:00:46,633 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e892275aa04716c37d873e5baba493e7: Waiting for close lock at 1676088046594Running coprocessor pre-close hooks at 1676088046594Disabling compacts and flushes for region at 1676088046594Disabling writes for close at 1676088046594Obtaining lock to block concurrent updates at 1676088046595 (+1 ms)Preparing flush snapshotting stores in e892275aa04716c37d873e5baba493e7 at 1676088046595Finished memstore snapshotting hbase:namespace,,1676088015613.e892275aa04716c37d873e5baba493e7., syncing WAL and waiting on mvcc, flushsize=dataSize=78, getHeapSize=472, getOffHeapSize=0, getCellsCount=2 at 1676088046595Flushing stores of hbase:namespace,,1676088015613.e892275aa04716c37d873e5baba493e7. at 1676088046599 (+4 ms)Flushing e892275aa04716c37d873e5baba493e7/info: creating writer at 1676088046599Flushing e892275aa04716c37d873e5baba493e7/info: appending metadata at 1676088046604 (+5 ms)Flushing e892275aa04716c37d873e5baba493e7/info: closing flushed file at 1676088046604Flushing e892275aa04716c37d873e5baba493e7/info: reopening flushed file at 1676088046625 (+21 ms)Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for e892275aa04716c37d873e5baba493e7 in 34ms, sequenceid=6, compaction requested=false at 1676088046628 (+3 ms)Writing region close event to WAL at 1676088046629 (+1 ms)Running coprocessor post-close hooks at 1676088046632 (+3 ms)Closed at 1676088046633 (+1 ms) 2023-02-11 04:00:46,633 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1676088015613.e892275aa04716c37d873e5baba493e7. 2023-02-11 04:00:46,638 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.01 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/meta/1588230740/.tmp/rep_barrier/74d4dd6a370a4bce810397a7fe24703e 2023-02-11 04:00:46,640 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 74d4dd6a370a4bce810397a7fe24703e 2023-02-11 04:00:46,650 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.75 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/meta/1588230740/.tmp/table/95641f0df4f5498ea16912c998d424e0 2023-02-11 04:00:46,653 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 95641f0df4f5498ea16912c998d424e0 2023-02-11 04:00:46,653 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/meta/1588230740/.tmp/info/330ed09142de4da98d3f2026b5dce8f8 as hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/meta/1588230740/info/330ed09142de4da98d3f2026b5dce8f8 2023-02-11 04:00:46,656 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 330ed09142de4da98d3f2026b5dce8f8 2023-02-11 04:00:46,656 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/meta/1588230740/info/330ed09142de4da98d3f2026b5dce8f8, entries=64, sequenceid=307, filesize=11.8 K 2023-02-11 04:00:46,656 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/meta/1588230740/.tmp/rep_barrier/74d4dd6a370a4bce810397a7fe24703e as hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/meta/1588230740/rep_barrier/74d4dd6a370a4bce810397a7fe24703e 2023-02-11 04:00:46,659 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 74d4dd6a370a4bce810397a7fe24703e 2023-02-11 04:00:46,659 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/meta/1588230740/rep_barrier/74d4dd6a370a4bce810397a7fe24703e, entries=18, sequenceid=307, filesize=7.0 K 2023-02-11 04:00:46,659 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/meta/1588230740/.tmp/table/95641f0df4f5498ea16912c998d424e0 as hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/meta/1588230740/table/95641f0df4f5498ea16912c998d424e0 2023-02-11 04:00:46,662 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 95641f0df4f5498ea16912c998d424e0 2023-02-11 04:00:46,662 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/meta/1588230740/table/95641f0df4f5498ea16912c998d424e0, entries=22, sequenceid=307, filesize=7.0 K 2023-02-11 04:00:46,662 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~137.71 KB/141010, heapSize ~212.43 KB/217528, currentSize=0 B/0 for 1588230740 in 61ms, sequenceid=307, compaction requested=false 2023-02-11 04:00:46,666 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/data/hbase/meta/1588230740/recovered.edits/310.seqid, newMaxSeqId=310, maxSeqId=1 2023-02-11 04:00:46,666 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-11 04:00:46,668 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-11 04:00:46,668 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676088046599Running coprocessor pre-close hooks at 1676088046599Disabling compacts and flushes for region at 1676088046599Disabling writes for close at 1676088046600 (+1 ms)Obtaining lock to block concurrent updates at 1676088046601 (+1 ms)Preparing flush snapshotting stores in 1588230740 at 1676088046601Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=141010, getHeapSize=217528, getOffHeapSize=0, getCellsCount=997 at 1676088046601Flushing stores of hbase:meta,,1.1588230740 at 1676088046603 (+2 ms)Flushing 1588230740/info: creating writer at 1676088046603Flushing 1588230740/info: appending metadata at 1676088046612 (+9 ms)Flushing 1588230740/info: closing flushed file at 1676088046612Flushing 1588230740/rep_barrier: creating writer at 1676088046628 (+16 ms)Flushing 1588230740/rep_barrier: appending metadata at 1676088046631 (+3 ms)Flushing 1588230740/rep_barrier: closing flushed file at 1676088046631Flushing 1588230740/table: creating writer at 1676088046640 (+9 ms)Flushing 1588230740/table: appending metadata at 1676088046642 (+2 ms)Flushing 1588230740/table: closing flushed file at 1676088046642Flushing 1588230740/info: reopening flushed file at 1676088046654 (+12 ms)Flushing 1588230740/rep_barrier: reopening flushed file at 1676088046657 (+3 ms)Flushing 1588230740/table: reopening flushed file at 1676088046660 (+3 ms)Finished flush of dataSize ~137.71 KB/141010, heapSize ~212.43 KB/217528, currentSize=0 B/0 for 1588230740 in 61ms, sequenceid=307, compaction requested=false at 1676088046662 (+2 ms)Writing region close event to WAL at 1676088046664 (+2 ms)Running coprocessor post-close hooks at 1676088046666 (+2 ms)Closed at 1676088046668 (+2 ms) 2023-02-11 04:00:46,668 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-11 04:00:46,699 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 04:00:46,699 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): regionserver:40829-0x1004189558d0002, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:46,699 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): regionserver:42385-0x1004189558d0003, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:46,699 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): regionserver:40829-0x1004189558d0002, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 04:00:46,699 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): regionserver:42385-0x1004189558d0003, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 04:00:46,699 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): regionserver:46851-0x1004189558d0001, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,46851,1676088013694 2023-02-11 04:00:46,699 ERROR [Listener at localhost/33753-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4377fbcd rejected from java.util.concurrent.ThreadPoolExecutor@4748677b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:00:46,700 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): regionserver:46851-0x1004189558d0001, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 04:00:46,700 ERROR [Listener at localhost/33753-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4f5765e0 rejected from java.util.concurrent.ThreadPoolExecutor@4748677b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:00:46,709 DEBUG [RS:2;jenkins-hbase4:42385] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/oldWALs 2023-02-11 04:00:46,710 INFO [RS:2;jenkins-hbase4:42385] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C42385%2C1676088013771:(num 1676088015249) 2023-02-11 04:00:46,710 DEBUG [RS:2;jenkins-hbase4:42385] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:00:46,710 INFO [RS:2;jenkins-hbase4:42385] regionserver.LeaseManager(133): Closed leases 2023-02-11 04:00:46,710 INFO [RS:2;jenkins-hbase4:42385] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-11 04:00:46,710 INFO [RS:2;jenkins-hbase4:42385] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-11 04:00:46,710 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-11 04:00:46,710 INFO [RS:2;jenkins-hbase4:42385] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-11 04:00:46,711 INFO [RS:2;jenkins-hbase4:42385] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-11 04:00:46,711 INFO [RS:2;jenkins-hbase4:42385] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:42385 2023-02-11 04:00:46,795 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,46851,1676088013694] 2023-02-11 04:00:46,796 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,46851,1676088013694; numProcessing=1 2023-02-11 04:00:46,797 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): regionserver:40829-0x1004189558d0002, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:46,797 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 04:00:46,797 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): regionserver:42385-0x1004189558d0003, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,42385,1676088013771 2023-02-11 04:00:46,797 ERROR [Listener at localhost/33753-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@40561a91 rejected from java.util.concurrent.ThreadPoolExecutor@44699643[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:00:46,798 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,46851,1676088013694 already deleted, retry=false 2023-02-11 04:00:46,798 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,46851,1676088013694 expired; onlineServers=2 2023-02-11 04:00:46,798 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,42385,1676088013771] 2023-02-11 04:00:46,798 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,42385,1676088013771; numProcessing=2 2023-02-11 04:00:46,800 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,42385,1676088013771 already deleted, retry=false 2023-02-11 04:00:46,800 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,42385,1676088013771 expired; onlineServers=1 2023-02-11 04:00:46,802 INFO [RS:1;jenkins-hbase4:40829] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,40829,1676088013739; all regions closed. 2023-02-11 04:00:46,806 DEBUG [RS:1;jenkins-hbase4:40829] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/oldWALs 2023-02-11 04:00:46,806 INFO [RS:1;jenkins-hbase4:40829] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C40829%2C1676088013739.meta:.meta(num 1676088015361) 2023-02-11 04:00:46,810 DEBUG [RS:1;jenkins-hbase4:40829] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/oldWALs 2023-02-11 04:00:46,810 INFO [RS:1;jenkins-hbase4:40829] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C40829%2C1676088013739:(num 1676088015246) 2023-02-11 04:00:46,811 DEBUG [RS:1;jenkins-hbase4:40829] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:00:46,811 INFO [RS:1;jenkins-hbase4:40829] regionserver.LeaseManager(133): Closed leases 2023-02-11 04:00:46,811 INFO [RS:1;jenkins-hbase4:40829] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-11 04:00:46,811 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-11 04:00:46,812 INFO [RS:1;jenkins-hbase4:40829] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:40829 2023-02-11 04:00:46,813 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): regionserver:40829-0x1004189558d0002, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,40829,1676088013739 2023-02-11 04:00:46,813 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 04:00:46,813 ERROR [Listener at localhost/33753-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@60deb4a7 rejected from java.util.concurrent.ThreadPoolExecutor@64e12c64[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:00:46,814 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,40829,1676088013739] 2023-02-11 04:00:46,815 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,40829,1676088013739; numProcessing=3 2023-02-11 04:00:46,816 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,40829,1676088013739 already deleted, retry=false 2023-02-11 04:00:46,816 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,40829,1676088013739 expired; onlineServers=0 2023-02-11 04:00:46,816 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,41713,1676088012566' ***** 2023-02-11 04:00:46,816 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-02-11 04:00:46,816 DEBUG [M:0;jenkins-hbase4:41713] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@71ddff3d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-11 04:00:46,816 INFO [M:0;jenkins-hbase4:41713] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,41713,1676088012566 2023-02-11 04:00:46,816 INFO [M:0;jenkins-hbase4:41713] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,41713,1676088012566; all regions closed. 2023-02-11 04:00:46,816 DEBUG [M:0;jenkins-hbase4:41713] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 04:00:46,817 DEBUG [M:0;jenkins-hbase4:41713] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-11 04:00:46,817 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-11 04:00:46,817 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676088014853] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676088014853,5,FailOnTimeoutGroup] 2023-02-11 04:00:46,817 DEBUG [M:0;jenkins-hbase4:41713] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-11 04:00:46,817 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676088014852] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676088014852,5,FailOnTimeoutGroup] 2023-02-11 04:00:46,817 INFO [M:0;jenkins-hbase4:41713] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-11 04:00:46,817 INFO [M:0;jenkins-hbase4:41713] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-11 04:00:46,817 INFO [M:0;jenkins-hbase4:41713] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-02-11 04:00:46,818 DEBUG [M:0;jenkins-hbase4:41713] master.HMaster(1502): Stopping service threads 2023-02-11 04:00:46,818 INFO [M:0;jenkins-hbase4:41713] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-11 04:00:46,818 ERROR [M:0;jenkins-hbase4:41713] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[HFileArchiver-3,5,PEWorkerGroup] Thread[HFileArchiver-4,5,PEWorkerGroup] Thread[HFileArchiver-5,5,PEWorkerGroup] Thread[HFileArchiver-6,5,PEWorkerGroup] Thread[HFileArchiver-7,5,PEWorkerGroup] Thread[HFileArchiver-8,5,PEWorkerGroup] 2023-02-11 04:00:46,818 INFO [M:0;jenkins-hbase4:41713] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-11 04:00:46,819 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-11 04:00:46,819 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-11 04:00:46,819 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 04:00:46,819 DEBUG [M:0;jenkins-hbase4:41713] zookeeper.ZKUtil(398): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-11 04:00:46,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 04:00:46,819 WARN [M:0;jenkins-hbase4:41713] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-11 04:00:46,819 INFO [M:0;jenkins-hbase4:41713] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-11 04:00:46,819 INFO [M:0;jenkins-hbase4:41713] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-11 04:00:46,819 DEBUG [M:0;jenkins-hbase4:41713] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-11 04:00:46,820 INFO [M:0;jenkins-hbase4:41713] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 04:00:46,820 DEBUG [M:0;jenkins-hbase4:41713] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 04:00:46,820 DEBUG [M:0;jenkins-hbase4:41713] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-11 04:00:46,820 DEBUG [M:0;jenkins-hbase4:41713] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 04:00:46,820 INFO [M:0;jenkins-hbase4:41713] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=845.84 KB heapSize=1.01 MB 2023-02-11 04:00:46,831 INFO [M:0;jenkins-hbase4:41713] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=845.84 KB at sequenceid=2120 (bloomFilter=true), to=hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/c579724ebc15445882abf1c1f2ff8b17 2023-02-11 04:00:46,834 INFO [M:0;jenkins-hbase4:41713] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c579724ebc15445882abf1c1f2ff8b17 2023-02-11 04:00:46,836 DEBUG [M:0;jenkins-hbase4:41713] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/c579724ebc15445882abf1c1f2ff8b17 as hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/c579724ebc15445882abf1c1f2ff8b17 2023-02-11 04:00:46,838 INFO [M:0;jenkins-hbase4:41713] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c579724ebc15445882abf1c1f2ff8b17 2023-02-11 04:00:46,839 INFO [M:0;jenkins-hbase4:41713] regionserver.HStore(1080): Added hdfs://localhost:37271/user/jenkins/test-data/a800a97a-fe96-7011-0371-90c87ad1afe6/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/c579724ebc15445882abf1c1f2ff8b17, entries=282, sequenceid=2120, filesize=30.9 K 2023-02-11 04:00:46,839 INFO [M:0;jenkins-hbase4:41713] regionserver.HRegion(2947): Finished flush of dataSize ~845.84 KB/866139, heapSize ~1.01 MB/1062976, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 19ms, sequenceid=2120, compaction requested=false 2023-02-11 04:00:46,840 INFO [M:0;jenkins-hbase4:41713] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 04:00:46,840 DEBUG [M:0;jenkins-hbase4:41713] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676088046819Disabling compacts and flushes for region at 1676088046819Disabling writes for close at 1676088046820 (+1 ms)Obtaining lock to block concurrent updates at 1676088046820Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676088046820Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=866139, getHeapSize=1062976, getOffHeapSize=0, getCellsCount=2607 at 1676088046820Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676088046820Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676088046821 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676088046825 (+4 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676088046825Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676088046836 (+11 ms)Finished flush of dataSize ~845.84 KB/866139, heapSize ~1.01 MB/1062976, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 19ms, sequenceid=2120, compaction requested=false at 1676088046839 (+3 ms)Writing region close event to WAL at 1676088046840 (+1 ms)Closed at 1676088046840 2023-02-11 04:00:46,843 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-11 04:00:46,843 INFO [M:0;jenkins-hbase4:41713] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-11 04:00:46,844 INFO [M:0;jenkins-hbase4:41713] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:41713 2023-02-11 04:00:46,845 DEBUG [M:0;jenkins-hbase4:41713] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,41713,1676088012566 already deleted, retry=false 2023-02-11 04:00:46,989 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 04:00:46,989 INFO [M:0;jenkins-hbase4:41713] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,41713,1676088012566; zookeeper connection closed. 2023-02-11 04:00:46,989 ERROR [Listener at localhost/33753-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2ac3fdd1 rejected from java.util.concurrent.ThreadPoolExecutor@703aed5b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 28] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:00:46,990 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): master:41713-0x1004189558d0000, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 04:00:46,990 ERROR [Listener at localhost/33753-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@79d19806 rejected from java.util.concurrent.ThreadPoolExecutor@703aed5b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 28] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:00:47,089 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): regionserver:40829-0x1004189558d0002, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 04:00:47,089 INFO [RS:1;jenkins-hbase4:40829] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,40829,1676088013739; zookeeper connection closed. 2023-02-11 04:00:47,090 ERROR [Listener at localhost/33753-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5666c53 rejected from java.util.concurrent.ThreadPoolExecutor@64e12c64[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:00:47,090 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): regionserver:40829-0x1004189558d0002, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 04:00:47,090 ERROR [Listener at localhost/33753-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7d6717bd rejected from java.util.concurrent.ThreadPoolExecutor@64e12c64[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:00:47,090 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@150514bf] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@150514bf 2023-02-11 04:00:47,190 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): regionserver:42385-0x1004189558d0003, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 04:00:47,190 INFO [RS:2;jenkins-hbase4:42385] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,42385,1676088013771; zookeeper connection closed. 2023-02-11 04:00:47,190 ERROR [Listener at localhost/33753-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@40520dd1 rejected from java.util.concurrent.ThreadPoolExecutor@44699643[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:00:47,190 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): regionserver:42385-0x1004189558d0003, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 04:00:47,190 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@cea481d] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@cea481d 2023-02-11 04:00:47,190 ERROR [Listener at localhost/33753-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@38f35611 rejected from java.util.concurrent.ThreadPoolExecutor@44699643[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:00:47,290 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): regionserver:46851-0x1004189558d0001, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 04:00:47,290 INFO [RS:0;jenkins-hbase4:46851] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,46851,1676088013694; zookeeper connection closed. 2023-02-11 04:00:47,290 ERROR [Listener at localhost/33753-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@37759ce0 rejected from java.util.concurrent.ThreadPoolExecutor@4748677b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:00:47,290 DEBUG [Listener at localhost/33753-EventThread] zookeeper.ZKWatcher(600): regionserver:46851-0x1004189558d0001, quorum=127.0.0.1:63425, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 04:00:47,290 ERROR [Listener at localhost/33753-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@16e91f70 rejected from java.util.concurrent.ThreadPoolExecutor@4748677b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 04:00:47,290 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@c428137] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@c428137 2023-02-11 04:00:47,290 INFO [Listener at localhost/33753] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-02-11 04:00:47,291 WARN [Listener at localhost/33753] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-11 04:00:47,294 INFO [Listener at localhost/33753] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-11 04:00:47,400 WARN [BP-184038972-172.31.14.131-1676088009489 heartbeating to localhost/127.0.0.1:37271] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-11 04:00:47,401 WARN [BP-184038972-172.31.14.131-1676088009489 heartbeating to localhost/127.0.0.1:37271] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-184038972-172.31.14.131-1676088009489 (Datanode Uuid 9cf11592-7e5f-4e4e-a42e-40b00afd1a4c) service to localhost/127.0.0.1:37271 2023-02-11 04:00:47,402 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/cluster_b673684c-6f28-f6ab-2a99-eadf0e1a7de2/dfs/data/data5/current/BP-184038972-172.31.14.131-1676088009489] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 04:00:47,403 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/cluster_b673684c-6f28-f6ab-2a99-eadf0e1a7de2/dfs/data/data6/current/BP-184038972-172.31.14.131-1676088009489] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 04:00:47,404 WARN [Listener at localhost/33753] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-11 04:00:47,406 INFO [Listener at localhost/33753] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-11 04:00:47,420 WARN [BP-184038972-172.31.14.131-1676088009489 heartbeating to localhost/127.0.0.1:37271] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-184038972-172.31.14.131-1676088009489 (Datanode Uuid e749fdb3-195e-48c9-9e67-ba30e7a3d54e) service to localhost/127.0.0.1:37271 2023-02-11 04:00:47,421 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/cluster_b673684c-6f28-f6ab-2a99-eadf0e1a7de2/dfs/data/data3/current/BP-184038972-172.31.14.131-1676088009489] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 04:00:47,421 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/cluster_b673684c-6f28-f6ab-2a99-eadf0e1a7de2/dfs/data/data4/current/BP-184038972-172.31.14.131-1676088009489] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 04:00:47,512 WARN [Listener at localhost/33753] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-11 04:00:47,514 INFO [Listener at localhost/33753] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-11 04:00:47,618 WARN [BP-184038972-172.31.14.131-1676088009489 heartbeating to localhost/127.0.0.1:37271] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-11 04:00:47,618 WARN [BP-184038972-172.31.14.131-1676088009489 heartbeating to localhost/127.0.0.1:37271] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-184038972-172.31.14.131-1676088009489 (Datanode Uuid 45440c1d-4b08-4665-a898-afaa9522b4b6) service to localhost/127.0.0.1:37271 2023-02-11 04:00:47,618 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/cluster_b673684c-6f28-f6ab-2a99-eadf0e1a7de2/dfs/data/data1/current/BP-184038972-172.31.14.131-1676088009489] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 04:00:47,619 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a4311e4d-5a1b-14b8-1b03-9cdea930efb3/cluster_b673684c-6f28-f6ab-2a99-eadf0e1a7de2/dfs/data/data2/current/BP-184038972-172.31.14.131-1676088009489] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 04:00:47,645 INFO [Listener at localhost/33753] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-11 04:00:47,777 INFO [Listener at localhost/33753] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-02-11 04:00:47,874 INFO [Listener at localhost/33753] hbase.HBaseTestingUtility(1293): Minicluster is down